[jira] [Commented] (CONNECTORS-1495) Brand new website
[ https://issues.apache.org/jira/browse/CONNECTORS-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798551#comment-17798551 ] Karl Wright commented on CONNECTORS-1495: - Nice work!! That was fast, and I agree with putting all the materials in the repo for it. Have you thought about licensing of those materials? This would be the time... > Brand new website > - > > Key: CONNECTORS-1495 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1495 > Project: ManifoldCF > Issue Type: New Feature > Components: Site >Affects Versions: ManifoldCF 2.9.1 >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Fix For: ManifoldCF next > > Attachments: ManifoldCF-FluidoSkin.png, PDF-Rendition-1.png, > PDF-Rendition-2.png, Website - status - 20180510-2.png, Website - status - > 20180510.png > > Original Estimate: 480h > Remaining Estimate: 480h > > The community decided to work on a brand new website: > [http://mail-archives.apache.org/mod_mbox/manifoldcf-dev/201712.mbox/%3CCAHVHQx8odjgXMw%3DnhmSeDt0pYOUd0j%2BtkmMNtFnCJvHFcZwyEg%40mail.gmail.com%3E] > The proposed technology is Jekyll but we have also to decide the website > template to use. > [~kamaci] suggested the [Apache CloudStack|https://cloudstack.apache.org/] > template. > [~molgun] proposed this approach: > # Find a modern new static site generator like Jekyll [1] > # Create a template > # Start to use it in a specific path like > [https://manifoldcf.apache.org/*new*] > # Migrate our Forrest xml's to Markdown (we can automate this somehow) > # Start to serve our new site on root path > [1] [https://jekyllrb.com/docs/home/] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1495) Brand new website
[ https://issues.apache.org/jira/browse/CONNECTORS-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796720#comment-17796720 ] Karl Wright commented on CONNECTORS-1495: - Hi [~piergiorgioluc...@gmail.com], I see no actual HTML pages for your new site. What am I missing? To publish this, all that is needed is to: (1) svn remove all directories and files under the svn pub/sub URL (2) Copy the new site in (3) Use svn add * to add all the directories and files (4) svn commit publishes it. The svn url is URL: https://svn.apache.org/repos/asf/manifoldcf/site/publish > Brand new website > - > > Key: CONNECTORS-1495 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1495 > Project: ManifoldCF > Issue Type: New Feature > Components: Site >Affects Versions: ManifoldCF 2.9.1 >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi >Priority: Major > Fix For: ManifoldCF next > > Attachments: ManifoldCF-FluidoSkin.png, PDF-Rendition-1.png, > PDF-Rendition-2.png, Website - status - 20180510-2.png, Website - status - > 20180510.png > > Original Estimate: 480h > Remaining Estimate: 480h > > The community decided to work on a brand new website: > [http://mail-archives.apache.org/mod_mbox/manifoldcf-dev/201712.mbox/%3CCAHVHQx8odjgXMw%3DnhmSeDt0pYOUd0j%2BtkmMNtFnCJvHFcZwyEg%40mail.gmail.com%3E] > The proposed technology is Jekyll but we have also to decide the website > template to use. > [~kamaci] suggested the [Apache CloudStack|https://cloudstack.apache.org/] > template. > [~molgun] proposed this approach: > # Find a modern new static site generator like Jekyll [1] > # Create a template > # Start to use it in a specific path like > [https://manifoldcf.apache.org/*new*] > # Migrate our Forrest xml's to Markdown (we can automate this somehow) > # Start to serve our new site on root path > [1] [https://jekyllrb.com/docs/home/] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CONNECTORS-1746) Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling become extremely slow.
[ https://issues.apache.org/jira/browse/CONNECTORS-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1746. - Fix Version/s: ManifoldCF 2.25 Resolution: Fixed > Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling > become extremely slow. > -- > > Key: CONNECTORS-1746 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1746 > Project: ManifoldCF > Issue Type: Improvement > Components: Web connector > Environment: Using ManifoldCF 2.24 with PostgreSQL 12.14 as the > database. >Reporter: Mingchun Zhao >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.25 > > Attachments: DBInterfacePostgreSQL.java.patch > > > Sometimes, the crawling does not process any documents for a while and there > is nothing logged about long-running queries. The performance can be restored > by firing the 'ANALYZE' command manually. It seems that a bad query plan > caused this performance problem. > Therefore, in addition to the current configuration parameter > 'org.apache.manifoldcf.db.postgres.analyze.', it is considered > necessary to execute the 'ANALYZE' even in the following situations. > 1. When the number of records in the table exceeds the number required for > creating a execution plan after the job starts. > 2. When the crawling performance slows down. For example, if the processing > rate of documents drops below a specified threshold. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1747) Add a property to disable logging hop count to database
[ https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17726821#comment-17726821 ] Karl Wright commented on CONNECTORS-1747: - I am away from fast internet until Monday. I will put up a release candidate then and call a vote. > Add a property to disable logging hop count to database > --- > > Key: CONNECTORS-1747 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1747 > Project: ManifoldCF > Issue Type: Improvement >Reporter: Mingchun Zhao >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.25 > > Attachments: CONNECTORS-1747.patch > > > If we do not require “Hop Filters“ feature, we need to consider to disable > logging records related to hopcount to database like "intrinsiclink" and > "hopcount" tables. This can increase throughput and reduce the rate of growth > of the database. > I will try to create a patch for this. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1747) Add a property to disable logging hop count to database
[ https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725989#comment-17725989 ] Karl Wright commented on CONNECTORS-1747: - I can put up a release candidate easily enough; however, it may be hard to get a voting quorum. That's the issue these days in this project. > Add a property to disable logging hop count to database > --- > > Key: CONNECTORS-1747 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1747 > Project: ManifoldCF > Issue Type: Improvement >Reporter: Mingchun Zhao >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.25 > > Attachments: CONNECTORS-1747.patch > > > If we do not require “Hop Filters“ feature, we need to consider to disable > logging records related to hopcount to database like "intrinsiclink" and > "hopcount" tables. This can increase throughput and reduce the rate of growth > of the database. > I will try to create a patch for this. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CONNECTORS-1747) Add a property to disable logging hop count to database
[ https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright updated CONNECTORS-1747: Fix Version/s: ManifoldCF 2.25 (was: ManifoldCF next) > Add a property to disable logging hop count to database > --- > > Key: CONNECTORS-1747 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1747 > Project: ManifoldCF > Issue Type: Improvement >Reporter: Mingchun Zhao >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.25 > > Attachments: CONNECTORS-1747.patch > > > If we do not require “Hop Filters“ feature, we need to consider to disable > logging records related to hopcount to database like "intrinsiclink" and > "hopcount" tables. This can increase throughput and reduce the rate of growth > of the database. > I will try to create a patch for this. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CONNECTORS-1747) Add a property to disable logging hop count to database
[ https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1747. - Fix Version/s: ManifoldCF next Resolution: Fixed r1910036 > Add a property to disable logging hop count to database > --- > > Key: CONNECTORS-1747 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1747 > Project: ManifoldCF > Issue Type: Improvement >Reporter: Mingchun Zhao >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF next > > Attachments: CONNECTORS-1747.patch > > > If we do not require “Hop Filters“ feature, we need to consider to disable > logging records related to hopcount to database like "intrinsiclink" and > "hopcount" tables. This can increase throughput and reduce the rate of growth > of the database. > I will try to create a patch for this. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1747) Add a property to disable logging hop count to database
[ https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725921#comment-17725921 ] Karl Wright commented on CONNECTORS-1747: - This looks good. I'll try to commit it tonight. > Add a property to disable logging hop count to database > --- > > Key: CONNECTORS-1747 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1747 > Project: ManifoldCF > Issue Type: Improvement >Reporter: Mingchun Zhao >Assignee: Karl Wright >Priority: Major > Attachments: CONNECTORS-1747.patch > > > If we do not require “Hop Filters“ feature, we need to consider to disable > logging records related to hopcount to database like "intrinsiclink" and > "hopcount" tables. This can increase throughput and reduce the rate of growth > of the database. > I will try to create a patch for this. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1747) Add a property to disable logging hop count to database
[ https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724654#comment-17724654 ] Karl Wright commented on CONNECTORS-1747: - Hi - so just to be clear, what you need to do here is: (1) Introduce a property, as you have done, that disables support for hopcount handling completely. It obviously should be a global cluster property, not a local one. (2) When that property is set, the HopCount.java class should never record anything in the intrinsicLinks or HopCount tables at all. (3) When that property is set, the Hopcount tab should not appear in the UI for any job. > Add a property to disable logging hop count to database > --- > > Key: CONNECTORS-1747 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1747 > Project: ManifoldCF > Issue Type: Improvement >Reporter: Mingchun Zhao >Assignee: Karl Wright >Priority: Major > Attachments: JobManager.java.patch > > > If we do not require “Hop Filters“ feature, we need to consider to disable > logging records related to hopcount to database like "intrinsiclink" and > "hopcount" tables. This can increase throughput and reduce the rate of growth > of the database. > I will try to create a patch for this. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1747) Add a property to disable logging hop count to database
[ https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724652#comment-17724652 ] Karl Wright commented on CONNECTORS-1747: - [~mingchun.zhao], it will be necessary to also disable the hopcount tab for all jobs entirely if you set this flag, since essentially the installation no longer can track hopcount at all. Please include that in your commit, thanks. > Add a property to disable logging hop count to database > --- > > Key: CONNECTORS-1747 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1747 > Project: ManifoldCF > Issue Type: Improvement >Reporter: Mingchun Zhao >Assignee: Karl Wright >Priority: Major > Attachments: JobManager.java.patch > > > If we do not require “Hop Filters“ feature, we need to consider to disable > logging records related to hopcount to database like "intrinsiclink" and > "hopcount" tables. This can increase throughput and reduce the rate of growth > of the database. > I will try to create a patch for this. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (CONNECTORS-1747) Add a property to disable logging hop count to database
[ https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1747: --- Assignee: Karl Wright > Add a property to disable logging hop count to database > --- > > Key: CONNECTORS-1747 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1747 > Project: ManifoldCF > Issue Type: Improvement >Reporter: Mingchun Zhao >Assignee: Karl Wright >Priority: Major > Attachments: JobManager.java.patch > > > If we do not require “Hop Filters“ feature, we need to consider to disable > logging records related to hopcount to database like "intrinsiclink" and > "hopcount" tables. This can increase throughput and reduce the rate of growth > of the database. > I will try to create a patch for this. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1746) Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling become extremely slow.
[ https://issues.apache.org/jira/browse/CONNECTORS-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722261#comment-17722261 ] Karl Wright commented on CONNECTORS-1746: - Patch committed: r1909780 > Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling > become extremely slow. > -- > > Key: CONNECTORS-1746 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1746 > Project: ManifoldCF > Issue Type: Improvement > Components: Web connector > Environment: Using ManifoldCF 2.24 with PostgreSQL 12.14 as the > database. >Reporter: Mingchun Zhao >Assignee: Karl Wright >Priority: Major > Attachments: DBInterfacePostgreSQL.java.patch > > > Sometimes, the crawling does not process any documents for a while and there > is nothing logged about long-running queries. The performance can be restored > by firing the 'ANALYZE' command manually. It seems that a bad query plan > caused this performance problem. > Therefore, in addition to the current configuration parameter > 'org.apache.manifoldcf.db.postgres.analyze.', it is considered > necessary to execute the 'ANALYZE' even in the following situations. > 1. When the number of records in the table exceeds the number required for > creating a execution plan after the job starts. > 2. When the crawling performance slows down. For example, if the processing > rate of documents drops below a specified threshold. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1740) Solr 9 output connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712329#comment-17712329 ] Karl Wright commented on CONNECTORS-1740: - It looks like there's an existing and unresolved Zookeeper issue for this: https://issues.apache.org/jira/browse/ZOOKEEPER-3828 Unfortunately this makes going to Solr 9 a no-go, unless we can we use an older version of Zookeeper without this problem. Have you tried that? > Solr 9 output connector > --- > > Key: CONNECTORS-1740 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1740 > Project: ManifoldCF > Issue Type: Improvement > Components: Lucene/SOLR connector >Affects Versions: ManifoldCF 2.23 >Reporter: Julien Massiera >Assignee: Julien Massiera >Priority: Major > > The current Solr output connector is not compatible with Solr 9.x > We need to update the connector with SolrJ 9 and make sure that the custom > code (multipart post requests, basic/preemptive auth) is still required, and, > in case it is, port it ! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1740) Solr 9 output connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712327#comment-17712327
]
Karl Wright commented on CONNECTORS-1740:
-
Zookeeper tests are failing because Zookeeper is not doing what it is supposed
to. Basically, sessions are being destroyed so fast that the test cannot
actually do anything useful:
{code}
[junit] org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client
session timed out, have not heard from server in 2007ms for session id 0x0
[junit] at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1250)
[junit] [reader] INFO org.apache.zookeeper.ZooKeeper - Session: 0x0 closed
[junit] [reader-EventThread] INFO org.apache.zookeeper.ClientCnxn -
EventThread shut down for session: 0x0
[junit] [reader] INFO org.apache.zookeeper.ZooKeeper - Initiating client
connection, connectString=localhost:8348 sessionTimeout=2000
watcher=org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection$ZooKeeperWatcher@45c66a27
[junit] [reader] INFO org.apache.zookeeper.ClientCnxnSocket -
jute.maxbuffer value is 1048575 Bytes
[junit] [reader] INFO org.apache.zookeeper.ClientCnxn -
zookeeper.request.timeout value is 0. feature enabled=false
[junit] [reader-SendThread(localhost:8348)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
localhost/127.0.0.1:8348.
[junit] [reader-SendThread(localhost:8348)] INFO
org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to
authenticate using SASL (unknown error)
[junit] [reader-SendThread(localhost:8348)] WARN
org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from
server in 2008ms for session id 0x0
[junit] [reader-SendThread(localhost:8348)] WARN
org.apache.zookeeper.ClientCnxn - An exception was thrown while closing send
thread for session 0x0.
[junit] org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client
session timed out, have not heard from server in 2008ms for session id 0x0
{code}
This is, of course, fatal to ManifoldCF's use of Zookeeper for synchronization.
So we need to figure out how to work around this new "feature", or the whole
project is a no-go.
> Solr 9 output connector
> ---
>
> Key: CONNECTORS-1740
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1740
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Lucene/SOLR connector
>Affects Versions: ManifoldCF 2.23
>Reporter: Julien Massiera
>Assignee: Julien Massiera
>Priority: Major
>
> The current Solr output connector is not compatible with Solr 9.x
> We need to update the connector with SolrJ 9 and make sure that the custom
> code (multipart post requests, basic/preemptive auth) is still required, and,
> in case it is, port it !
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1740) Solr 9 output connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712321#comment-17712321
]
Karl Wright commented on CONNECTORS-1740:
-
Turns out that this update also will require we move permanently from Java 8 to
Java 11:
{code}
compile-connector:
[javac] Compiling 17 source files to
C:\wip\mcf\CONNECTORS-1740\connectors\solr\build\connector\classes
[javac]
C:\wip\mcf\CONNECTORS-1740\connectors\solr\connector\src\main\java\org\apache\manifoldcf\agents\output\solr\ModifiedHttp2SolrClient.java:3:
error: cannot access Utils
[javac] import static org.apache.solr.common.util.Utils.getObjectByPath;
[javac] ^
[javac] bad class file:
C:\wip\mcf\CONNECTORS-1740\lib\solr-solrj-9.1.0.jar(org/apache/solr/common/util/Utils.class)
[javac] class file has wrong version 55.0, should be 52.0
[javac] Please remove or make sure it appears in the correct
subdirectory of the classpath.
{code}
> Solr 9 output connector
> ---
>
> Key: CONNECTORS-1740
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1740
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Lucene/SOLR connector
>Affects Versions: ManifoldCF 2.23
>Reporter: Julien Massiera
>Assignee: Julien Massiera
>Priority: Major
>
> The current Solr output connector is not compatible with Solr 9.x
> We need to update the connector with SolrJ 9 and make sure that the custom
> code (multipart post requests, basic/preemptive auth) is still required, and,
> in case it is, port it !
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1740) Solr 9 output connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711593#comment-17711593 ] Karl Wright commented on CONNECTORS-1740: - [~julienFL], I'll look at the zookeeper changes. > Solr 9 output connector > --- > > Key: CONNECTORS-1740 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1740 > Project: ManifoldCF > Issue Type: Improvement > Components: Lucene/SOLR connector >Affects Versions: ManifoldCF 2.23 >Reporter: Julien Massiera >Assignee: Julien Massiera >Priority: Major > > The current Solr output connector is not compatible with Solr 9.x > We need to update the connector with SolrJ 9 and make sure that the custom > code (multipart post requests, basic/preemptive auth) is still required, and, > in case it is, port it ! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (CONNECTORS-1740) Solr 9 output connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1740: --- Assignee: Julien Massiera > Solr 9 output connector > --- > > Key: CONNECTORS-1740 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1740 > Project: ManifoldCF > Issue Type: Improvement > Components: Lucene/SOLR connector >Affects Versions: ManifoldCF 2.23 >Reporter: Julien Massiera >Assignee: Julien Massiera >Priority: Major > > The current Solr output connector is not compatible with Solr 9.x > We need to update the connector with SolrJ 9 and make sure that the custom > code (multipart post requests, basic/preemptive auth) is still required, and, > in case it is, port it ! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (CONNECTORS-1743) The Solr Output Connector should retry on a 502 Bad Gateway or 503 Service Unavailable
[ https://issues.apache.org/jira/browse/CONNECTORS-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1743: --- Assignee: Karl Wright > The Solr Output Connector should retry on a 502 Bad Gateway or 503 Service > Unavailable > -- > > Key: CONNECTORS-1743 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1743 > Project: ManifoldCF > Issue Type: Improvement >Reporter: Markus Günther >Assignee: Karl Wright >Priority: Major > Attachments: CONNECTORS-1743.patch > > > The Solr Output Connector (cf. HttpPoster) triggers a retry if the Solr > response returns with HTTP status code 500. This behavior should be extended > to include a 502 Bad Gateway as well as 503 Service Unavailable as well, as > these indicate transient error situations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CONNECTORS-1741) Documentation no longer available?
[ https://issues.apache.org/jira/browse/CONNECTORS-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1741. - Fix Version/s: ManifoldCF 2.24 Resolution: Fixed Was resolved by hand-pushing the required files to the mirror svn. > Documentation no longer available? > -- > > Key: CONNECTORS-1741 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1741 > Project: ManifoldCF > Issue Type: Wish > Components: Documentation >Reporter: Len G >Assignee: Karl Wright >Priority: Minor > Fix For: ManifoldCF 2.24 > > > The documentation on the website no longer seems to be available. When I try > to access the Release URLs at > [https://manifoldcf.apache.org/en_US/release-documentation.html], they are > not available (404 Not Found). > Am I missing something here or looking in the wrong location? They used to be > there. > Thanks > > __PRESENT > __PRESENT -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1741) Documentation no longer available?
[ https://issues.apache.org/jira/browse/CONNECTORS-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17643466#comment-17643466 ] Karl Wright commented on CONNECTORS-1741: - Infrastructure changes have apparently prevented us from publishing the release documentation on the site. Need to research why that is happening. Looked fine locally. > Documentation no longer available? > -- > > Key: CONNECTORS-1741 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1741 > Project: ManifoldCF > Issue Type: Wish > Components: Documentation >Reporter: Len G >Assignee: Karl Wright >Priority: Minor > > The documentation on the website no longer seems to be available. When I try > to access the Release URLs at > [https://manifoldcf.apache.org/en_US/release-documentation.html], they are > not available (404 Not Found). > Am I missing something here or looking in the wrong location? They used to be > there. > Thanks > > __PRESENT > __PRESENT -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (CONNECTORS-1741) Documentation no longer available?
[ https://issues.apache.org/jira/browse/CONNECTORS-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1741: --- Assignee: Karl Wright > Documentation no longer available? > -- > > Key: CONNECTORS-1741 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1741 > Project: ManifoldCF > Issue Type: Wish > Components: Documentation >Reporter: Len G >Assignee: Karl Wright >Priority: Minor > > The documentation on the website no longer seems to be available. When I try > to access the Release URLs at > [https://manifoldcf.apache.org/en_US/release-documentation.html], they are > not available (404 Not Found). > Am I missing something here or looking in the wrong location? They used to be > there. > Thanks > > __PRESENT > __PRESENT -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CONNECTORS-1738) Suggestion for adding function that allows setting timeout values for Elasticsearch Output Connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1738.
-
Fix Version/s: ManifoldCF 2.24
Resolution: Fixed
r1904741
> Suggestion for adding function that allows setting timeout values for
> Elasticsearch Output Connector
>
>
> Key: CONNECTORS-1738
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1738
> Project: ManifoldCF
> Issue Type: Improvement
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.24
>
> Attachments: EditConnection.PNG, ViewConnection.PNG, patch.txt
>
>
> Hi there,
> For Elasticsearch Output Connector, during use, I have exeperienced cases
> that required the values of *socketTimeout* and *connectionTimeout* to be
> increased.
> However, as those values are being hardcoded within the source code as
> 90(ms) and 6(ms) respectively, it is quite troublesome to update them
> in cases mentioned above.
> For this reason, instead of hardcoding, I think it would be better that the
> values of *socketTimeout* and *connectionTimeout* can be edited through
> WebUI, on the connection setting screen.
> In ManifoldCF, there are also a few other connectors that support setting
> *socketTimeout* and {*}connectionTimeout{*}, such as Generic, Confluence, etc.
> Therefore, I would like to suggest modifying the ElasticSearch Output
> Connector's source code to allow setting *socketTimeout* and
> *connectionTimeout* value when it is needed.
> h3. +*1. Connector Name*+
> ElasticSearch Output Connector
> h3. +*2. Improvement Detail*+
> On connection setting screen (WebUI), add handling method that enable value
> setting for *socketTimeout* and *connectionTimeout*
> ※The default value for *socketTimeout* and *connectionTimeout* are still
> 90 and 6 (ms) as they are.
> The connection setting screen will look like below:
> !EditConnection.PNG!
> h3. +*3. Suggested source code (based on release 2.22.1)*+
> Because the content is edited in many files & the number of LOC is quite
> large,
> I will attach the patch file here, please check it out.
> [^patch.txt]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CONNECTORS-1738) Suggestion for adding function that allows setting timeout values for Elasticsearch Output Connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1738:
---
Assignee: Karl Wright
> Suggestion for adding function that allows setting timeout values for
> Elasticsearch Output Connector
>
>
> Key: CONNECTORS-1738
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1738
> Project: ManifoldCF
> Issue Type: Improvement
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
> Attachments: EditConnection.PNG, ViewConnection.PNG, patch.txt
>
>
> Hi there,
> For Elasticsearch Output Connector, during use, I have exeperienced cases
> that required the values of *socketTimeout* and *connectionTimeout* to be
> increased.
> However, as those values are being hardcoded within the source code as
> 90(ms) and 6(ms) respectively, it is quite troublesome to update them
> in cases mentioned above.
> For this reason, instead of hardcoding, I think it would be better that the
> values of *socketTimeout* and *connectionTimeout* can be edited through
> WebUI, on the connection setting screen.
> In ManifoldCF, there are also a few other connectors that support setting
> *socketTimeout* and {*}connectionTimeout{*}, such as Generic, Confluence, etc.
> Therefore, I would like to suggest modifying the ElasticSearch Output
> Connector's source code to allow setting *socketTimeout* and
> *connectionTimeout* value when it is needed.
> h3. +*1. Connector Name*+
> ElasticSearch Output Connector
> h3. +*2. Improvement Detail*+
> On connection setting screen (WebUI), add handling method that enable value
> setting for *socketTimeout* and *connectionTimeout*
> ※The default value for *socketTimeout* and *connectionTimeout* are still
> 90 and 6 (ms) as they are.
> The connection setting screen will look like below:
> !EditConnection.PNG!
> h3. +*3. Suggested source code (based on release 2.22.1)*+
> Because the content is edited in many files & the number of LOC is quite
> large,
> I will attach the patch file here, please check it out.
> [^patch.txt]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CONNECTORS-1737) Suggestion for adding function for proxy configuration for connector of Confluence-V6
[ https://issues.apache.org/jira/browse/CONNECTORS-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1737. - Fix Version/s: ManifoldCF 2.24 Resolution: Fixed r1904412. > Suggestion for adding function for proxy configuration for connector of > Confluence-V6 > - > > Key: CONNECTORS-1737 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1737 > Project: ManifoldCF > Issue Type: Improvement >Reporter: Nguyen Huu Nhat >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.24 > > Attachments: EditConnection.png, ViewConnection.png, patch.txt > > > Hi there, > Currently, I am having a problem regarding the need for proxy information to > connect to Confluence Cloud (SaaS) from the Confluence connector. > As you know, we can use Confluence Server (self-built environment) or use > Confluence Cloud (SaaS). > Using Confluence Server, you can set up a server (for example, located in the > same place where ManifoldCF is running) so you may not need to use proxy > information for the connection. > However, for Confluence Cloud (SaaS), there will be cases where you need to > set proxy information to be able to connect. > I have checked in a few other connectors of ManifoldCF, there are a few > connectors that support setting proxy information for the connection, for > example: SharePoint, Jira, Slack, ... > Therefore, I would like to suggest editing Confluence-v6's sourcecode > (including Authority connector and Repository connector) so that we can have > an option to set proxy information to use for connection to Confluence when > needed. > h3. +*Connector Name*+ > - Confluence-v6 (including Authority connector and Repository connector) > h3. +*Improvement Detail*+ > - At the Confluence connection setting screen, add fields that allow the > user to set proxy information (eg: proxyProtocol, proxyHost, proxyPort..). > The proxy setting screen will look like below: > !EditConnection.png! > - When connecting (call method `connect()`), use the set field values as > proxy information. > ※ This improvement applies to both the Authority Connector and the Repository > Connector side. > h3. +*Suggested source code (based on release 2.22.1)*+ > Because the content is edited in many files and the number of LOC is quite > large, I will attach the patch file here, please check it. > [^patch.txt] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (CONNECTORS-1737) Suggestion for adding function for proxy configuration for connector of Confluence-V6
[ https://issues.apache.org/jira/browse/CONNECTORS-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1737: --- Assignee: Karl Wright > Suggestion for adding function for proxy configuration for connector of > Confluence-V6 > - > > Key: CONNECTORS-1737 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1737 > Project: ManifoldCF > Issue Type: Improvement >Reporter: Nguyen Huu Nhat >Assignee: Karl Wright >Priority: Major > Attachments: EditConnection.png, ViewConnection.png, patch.txt > > > Hi there, > Currently, I am having a problem regarding the need for proxy information to > connect to Confluence Cloud (SaaS) from the Confluence connector. > As you know, we can use Confluence Server (self-built environment) or use > Confluence Cloud (SaaS). > Using Confluence Server, you can set up a server (for example, located in the > same place where ManifoldCF is running) so you may not need to use proxy > information for the connection. > However, for Confluence Cloud (SaaS), there will be cases where you need to > set proxy information to be able to connect. > I have checked in a few other connectors of ManifoldCF, there are a few > connectors that support setting proxy information for the connection, for > example: SharePoint, Jira, Slack, ... > Therefore, I would like to suggest editing Confluence-v6's sourcecode > (including Authority connector and Repository connector) so that we can have > an option to set proxy information to use for connection to Confluence when > needed. > h3. +*Connector Name*+ > - Confluence-v6 (including Authority connector and Repository connector) > h3. +*Improvement Detail*+ > - At the Confluence connection setting screen, add fields that allow the > user to set proxy information (eg: proxyProtocol, proxyHost, proxyPort..). > The proxy setting screen will look like below: > !EditConnection.png! > - When connecting (call method `connect()`), use the set field values as > proxy information. > ※ This improvement applies to both the Authority Connector and the Repository > Connector side. > h3. +*Suggested source code (based on release 2.22.1)*+ > Because the content is edited in many files and the number of LOC is quite > large, I will attach the patch file here, please check it. > [^patch.txt] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CONNECTORS-1731) Suggestion for adding handling process for ManifoldCFException in Generic Repository Connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1731.
-
Fix Version/s: ManifoldCF next
Resolution: Fixed
> Suggestion for adding handling process for ManifoldCFException in Generic
> Repository Connector
> --
>
> Key: CONNECTORS-1731
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1731
> Project: ManifoldCF
> Issue Type: Improvement
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF next
>
> Attachments: patch.txt, patch_update.txt
>
>
> Hi there,
> I would like to suggest the following retry-related improvements:
> h3. +*1. Connector name*+
> Generic Repository Connector
> h3. +*2. Reasons for improvement*+
> In the process of ExecuteSeedingThread, DocumentVersionThread and
> ExecuteProcessThread, when HttpClient.execute is executed, there are chances
> that connection gets interrupted and connection error occurs (HTTP status
> code <> 200).
> When HTTP status code <> 200, exception of type ManifoldCFException will be
> thrown. However, there is no process to handle this, which leads to job being
> aborted.
> As errors relating to connection (HTTP status code <> 200) can be resolved
> automatically, retry should be added for cases like this.
> h3. +*3. Improvements*+
> Improvement includes the followings:
> * Adding method to handle retry for exception of type ManifoldCFException
> * Calling method to handle ManifoldCFException exception generated when
> executing in threads: ExecuteSeedingThread, DocumentVersionThread,
> ExecuteProcessThread
> h3. +*4. Suggested source code (based on release 2.22.1)*+
> * Adding method to handle retry for exception of type ManifoldCFException
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L1026]
> {code:java}
> /**
>* Function for handling ManifoldCFException exception caused by connection
> error.
>* In case of connection error, ServiceInterruption exception is thrown to
> perform retry.
>*
>* @param e ManifoldCFException
>* @throws ServiceInterruption
>*/
> protected static void handleManifoldCFException(ManifoldCFException e)
> throws ServiceInterruption {
> long currentTime = System.currentTimeMillis();
> throw new ServiceInterruption("Connection error: " + e.getMessage(), e,
> currentTime + 30L,
> currentTime + 3 * 60 * 6L, -1, false);
> }
> {code}
> * Calling method to handle ManifoldCFException exception generated when
> executing in threads: ExecuteSeedingThread, DocumentVersionThread,
> ExecuteProcessThread
> ** ExecuteSeedingThread
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L256]
> {code:java}
> } catch (InterruptedException e) {
> t.interrupt();
> throw new ManifoldCFException("Interrupted: " + e.getMessage(), e,
> ManifoldCFException.INTERRUPTED);
> + } catch (ManifoldCFException e) {
> + handleManifoldCFException(e);
> }
> return new Long(seedTime).toString();
> {code}
> ** DocumentVersionThread
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L304]
> {code:java}
> try {
> versions = versioningThread.finishUp();
> } catch (IOException ex) {
> handleIOException((IOException)ex);
> + } catch (ManifoldCFException ex) {
> + handleManifoldCFException(ex);
> } catch (InterruptedException ex) {
> throw new ManifoldCFException(ex.getMessage(), ex,
> ManifoldCFException.INTERRUPTED);
> }
> {code}
> ** ExecuteProcessThread
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L445]
> {code:java}
> } catch (InterruptedIOException e) {
> t.interrupt();
> throw new ManifoldCFException("Interrupted: " + e.getMessage(),
> e, ManifoldCFException.INTERRUPTED);
> } catch (IOException e) {
> handleIOException(e);
> + } catch (ManifoldCFException e) {
> + handleManifoldCFException(e);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1731) Suggestion for adding handling process for ManifoldCFException in Generic Repository Connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607635#comment-17607635
]
Karl Wright commented on CONNECTORS-1731:
-
r1904186
> Suggestion for adding handling process for ManifoldCFException in Generic
> Repository Connector
> --
>
> Key: CONNECTORS-1731
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1731
> Project: ManifoldCF
> Issue Type: Improvement
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
> Attachments: patch.txt, patch_update.txt
>
>
> Hi there,
> I would like to suggest the following retry-related improvements:
> h3. +*1. Connector name*+
> Generic Repository Connector
> h3. +*2. Reasons for improvement*+
> In the process of ExecuteSeedingThread, DocumentVersionThread and
> ExecuteProcessThread, when HttpClient.execute is executed, there are chances
> that connection gets interrupted and connection error occurs (HTTP status
> code <> 200).
> When HTTP status code <> 200, exception of type ManifoldCFException will be
> thrown. However, there is no process to handle this, which leads to job being
> aborted.
> As errors relating to connection (HTTP status code <> 200) can be resolved
> automatically, retry should be added for cases like this.
> h3. +*3. Improvements*+
> Improvement includes the followings:
> * Adding method to handle retry for exception of type ManifoldCFException
> * Calling method to handle ManifoldCFException exception generated when
> executing in threads: ExecuteSeedingThread, DocumentVersionThread,
> ExecuteProcessThread
> h3. +*4. Suggested source code (based on release 2.22.1)*+
> * Adding method to handle retry for exception of type ManifoldCFException
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L1026]
> {code:java}
> /**
>* Function for handling ManifoldCFException exception caused by connection
> error.
>* In case of connection error, ServiceInterruption exception is thrown to
> perform retry.
>*
>* @param e ManifoldCFException
>* @throws ServiceInterruption
>*/
> protected static void handleManifoldCFException(ManifoldCFException e)
> throws ServiceInterruption {
> long currentTime = System.currentTimeMillis();
> throw new ServiceInterruption("Connection error: " + e.getMessage(), e,
> currentTime + 30L,
> currentTime + 3 * 60 * 6L, -1, false);
> }
> {code}
> * Calling method to handle ManifoldCFException exception generated when
> executing in threads: ExecuteSeedingThread, DocumentVersionThread,
> ExecuteProcessThread
> ** ExecuteSeedingThread
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L256]
> {code:java}
> } catch (InterruptedException e) {
> t.interrupt();
> throw new ManifoldCFException("Interrupted: " + e.getMessage(), e,
> ManifoldCFException.INTERRUPTED);
> + } catch (ManifoldCFException e) {
> + handleManifoldCFException(e);
> }
> return new Long(seedTime).toString();
> {code}
> ** DocumentVersionThread
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L304]
> {code:java}
> try {
> versions = versioningThread.finishUp();
> } catch (IOException ex) {
> handleIOException((IOException)ex);
> + } catch (ManifoldCFException ex) {
> + handleManifoldCFException(ex);
> } catch (InterruptedException ex) {
> throw new ManifoldCFException(ex.getMessage(), ex,
> ManifoldCFException.INTERRUPTED);
> }
> {code}
> ** ExecuteProcessThread
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L445]
> {code:java}
> } catch (InterruptedIOException e) {
> t.interrupt();
> throw new ManifoldCFException("Interrupted: " + e.getMessage(),
> e, ManifoldCFException.INTERRUPTED);
> } catch (IOException e) {
> handleIOException(e);
> + } catch (ManifoldCFException e) {
> + handleManifoldCFException(e);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1731) Suggestion for adding handling process for ManifoldCFException in Generic Repository Connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607619#comment-17607619
]
Karl Wright commented on CONNECTORS-1731:
-
Patch applies but the code doesn't build. Here is the error:
{code}
[javac]
C:\wip\mcf\trunk\connectors\generic\connector\src\main\java\org\apache\manifoldcf\crawler\connectors\generic\GenericConnector.java:1044:
error: cannot find symbol
[javac] currentTime + timeToFail * 3 * 60 * 6L, -1, false);
[javac] ^
[javac] symbol: variable timeToFail
[javac] location: class GenericConnector
{code}
> Suggestion for adding handling process for ManifoldCFException in Generic
> Repository Connector
> --
>
> Key: CONNECTORS-1731
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1731
> Project: ManifoldCF
> Issue Type: Improvement
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
> Attachments: patch.txt
>
>
> Hi there,
> I would like to suggest the following retry-related improvements:
> h3. +*1. Connector name*+
> Generic Repository Connector
> h3. +*2. Reasons for improvement*+
> In the process of ExecuteSeedingThread, DocumentVersionThread and
> ExecuteProcessThread, when HttpClient.execute is executed, there are chances
> that connection gets interrupted and connection error occurs (HTTP status
> code <> 200).
> When HTTP status code <> 200, exception of type ManifoldCFException will be
> thrown. However, there is no process to handle this, which leads to job being
> aborted.
> As errors relating to connection (HTTP status code <> 200) can be resolved
> automatically, retry should be added for cases like this.
> h3. +*3. Improvements*+
> Improvement includes the followings:
> * Adding method to handle retry for exception of type ManifoldCFException
> * Calling method to handle ManifoldCFException exception generated when
> executing in threads: ExecuteSeedingThread, DocumentVersionThread,
> ExecuteProcessThread
> h3. +*4. Suggested source code (based on release 2.22.1)*+
> * Adding method to handle retry for exception of type ManifoldCFException
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L1026]
> {code:java}
> /**
>* Function for handling ManifoldCFException exception caused by connection
> error.
>* In case of connection error, ServiceInterruption exception is thrown to
> perform retry.
>*
>* @param e ManifoldCFException
>* @throws ServiceInterruption
>*/
> protected static void handleManifoldCFException(ManifoldCFException e)
> throws ServiceInterruption {
> long currentTime = System.currentTimeMillis();
> throw new ServiceInterruption("Connection error: " + e.getMessage(), e,
> currentTime + 30L,
> currentTime + timeToFail * 3 * 60 * 6L, -1, false);
> }
> {code}
> * Calling method to handle ManifoldCFException exception generated when
> executing in threads: ExecuteSeedingThread, DocumentVersionThread,
> ExecuteProcessThread
> ** ExecuteSeedingThread
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L256]
> {code:java}
> } catch (InterruptedException e) {
> t.interrupt();
> throw new ManifoldCFException("Interrupted: " + e.getMessage(), e,
> ManifoldCFException.INTERRUPTED);
> + } catch (ManifoldCFException e) {
> + handleManifoldCFException(e);
> }
> return new Long(seedTime).toString();
> {code}
> ** DocumentVersionThread
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L304]
> {code:java}
> try {
> versions = versioningThread.finishUp();
> } catch (IOException ex) {
> handleIOException((IOException)ex);
> + } catch (ManifoldCFException ex) {
> + handleManifoldCFException(ex);
> } catch (InterruptedException ex) {
> throw new ManifoldCFException(ex.getMessage(), ex,
> ManifoldCFException.INTERRUPTED);
> }
> {code}
> ** ExecuteProcessThread
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L445]
> {code:java}
> } catch (InterruptedIOException e) {
> t.interrupt();
> throw new ManifoldCFException("Interrupted: " + e.getMessage(),
> e, ManifoldCFException.INTERRUPTED);
> } catch
[jira] [Commented] (CONNECTORS-1731) Suggestion for adding handling process for ManifoldCFException in Generic Repository Connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607471#comment-17607471
]
Karl Wright commented on CONNECTORS-1731:
-
That sounds like a reasonable code cleanup. Can you submit this as a patch
file? Generate one by making your changes in an svn checkout and then doing:
svn diff >patch.txt
Attach patch.txt to this ticket. Thanks!
> Suggestion for adding handling process for ManifoldCFException in Generic
> Repository Connector
> --
>
> Key: CONNECTORS-1731
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1731
> Project: ManifoldCF
> Issue Type: Improvement
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
>
> Hi there,
> I would like to suggest the following retry-related improvements:
> h3. +*1. Connector name*+
> Generic Repository Connector
> h3. +*2. Reasons for improvement*+
> In the process of ExecuteSeedingThread, DocumentVersionThread and
> ExecuteProcessThread, when HttpClient.execute is executed, there are chances
> that connection gets interrupted and connection error occurs (HTTP status
> code <> 200).
> When HTTP status code <> 200, exception of type ManifoldCFException will be
> thrown. However, there is no process to handle this, which leads to job being
> aborted.
> As errors relating to connection (HTTP status code <> 200) can be resolved
> automatically, retry should be added for cases like this.
> h3. +*3. Improvements*+
> Improvement includes the followings:
> * Adding method to handle retry for exception of type ManifoldCFException
> * Calling method to handle ManifoldCFException exception generated when
> executing in threads: ExecuteSeedingThread, DocumentVersionThread,
> ExecuteProcessThread
> h3. +*4. Suggested source code (based on release 2.22.1)*+
> * Adding method to handle retry for exception of type ManifoldCFException
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L1026]
> {code:java}
> /**
>* Function for handling ManifoldCFException exception caused by connection
> error.
>* In case of connection error, ServiceInterruption exception is thrown to
> perform retry.
>*
>* @param e ManifoldCFException
>* @throws ServiceInterruption
>*/
> protected static void handleManifoldCFException(ManifoldCFException e)
> throws ServiceInterruption {
> long currentTime = System.currentTimeMillis();
> throw new ServiceInterruption("Connection error: " + e.getMessage(), e,
> currentTime + 30L,
> currentTime + timeToFail * 3 * 60 * 6L, -1, false);
> }
> {code}
> * Calling method to handle ManifoldCFException exception generated when
> executing in threads: ExecuteSeedingThread, DocumentVersionThread,
> ExecuteProcessThread
> ** ExecuteSeedingThread
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L256]
> {code:java}
> } catch (InterruptedException e) {
> t.interrupt();
> throw new ManifoldCFException("Interrupted: " + e.getMessage(), e,
> ManifoldCFException.INTERRUPTED);
> + } catch (ManifoldCFException e) {
> + handleManifoldCFException(e);
> }
> return new Long(seedTime).toString();
> {code}
> ** DocumentVersionThread
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L304]
> {code:java}
> try {
> versions = versioningThread.finishUp();
> } catch (IOException ex) {
> handleIOException((IOException)ex);
> + } catch (ManifoldCFException ex) {
> + handleManifoldCFException(ex);
> } catch (InterruptedException ex) {
> throw new ManifoldCFException(ex.getMessage(), ex,
> ManifoldCFException.INTERRUPTED);
> }
> {code}
> ** ExecuteProcessThread
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L445]
> {code:java}
> } catch (InterruptedIOException e) {
> t.interrupt();
> throw new ManifoldCFException("Interrupted: " + e.getMessage(),
> e, ManifoldCFException.INTERRUPTED);
> } catch (IOException e) {
> handleIOException(e);
> + } catch (ManifoldCFException e) {
> + handleManifoldCFException(e);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CONNECTORS-1731) Suggestion for adding handling process for ManifoldCFException in Generic Repository Connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1731:
---
Assignee: Karl Wright
> Suggestion for adding handling process for ManifoldCFException in Generic
> Repository Connector
> --
>
> Key: CONNECTORS-1731
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1731
> Project: ManifoldCF
> Issue Type: Improvement
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
>
> Hi there,
> I would like to suggest the following retry-related improvements:
> h3. +*1. Connector name*+
> Generic Repository Connector
> h3. +*2. Reasons for improvement*+
> In the process of ExecuteSeedingThread, DocumentVersionThread and
> ExecuteProcessThread, when HttpClient.execute is executed, there are chances
> that connection gets interrupted and connection error occurs (HTTP status
> code <> 200).
> When HTTP status code <> 200, exception of type ManifoldCFException will be
> thrown. However, there is no process to handle this, which leads to job being
> aborted.
> As errors relating to connection (HTTP status code <> 200) can be resolved
> automatically, retry should be added for cases like this.
> h3. +*3. Improvements*+
> Improvement includes the followings:
> * Adding method to handle retry for exception of type ManifoldCFException
> * Calling method to handle ManifoldCFException exception generated when
> executing in threads: ExecuteSeedingThread, DocumentVersionThread,
> ExecuteProcessThread
> h3. +*4. Suggested source code (based on release 2.22.1)*+
> * Adding method to handle retry for exception of type ManifoldCFException
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L1026]
> {code:java}
> /**
>* Function for handling ManifoldCFException exception caused by connection
> error.
>* In case of connection error, ServiceInterruption exception is thrown to
> perform retry.
>*
>* @param e ManifoldCFException
>* @throws ServiceInterruption
>*/
> protected static void handleManifoldCFException(ManifoldCFException e)
> throws ServiceInterruption {
> long currentTime = System.currentTimeMillis();
> throw new ServiceInterruption("Connection error: " + e.getMessage(), e,
> currentTime + 30L,
> currentTime + timeToFail * 3 * 60 * 6L, -1, false);
> }
> {code}
> * Calling method to handle ManifoldCFException exception generated when
> executing in threads: ExecuteSeedingThread, DocumentVersionThread,
> ExecuteProcessThread
> ** ExecuteSeedingThread
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L256]
> {code:java}
> } catch (InterruptedException e) {
> t.interrupt();
> throw new ManifoldCFException("Interrupted: " + e.getMessage(), e,
> ManifoldCFException.INTERRUPTED);
> + } catch (ManifoldCFException e) {
> + handleManifoldCFException(e);
> }
> return new Long(seedTime).toString();
> {code}
> ** DocumentVersionThread
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L304]
> {code:java}
> try {
> versions = versioningThread.finishUp();
> } catch (IOException ex) {
> handleIOException((IOException)ex);
> + } catch (ManifoldCFException ex) {
> + handleManifoldCFException(ex);
> } catch (InterruptedException ex) {
> throw new ManifoldCFException(ex.getMessage(), ex,
> ManifoldCFException.INTERRUPTED);
> }
> {code}
> ** ExecuteProcessThread
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L445]
> {code:java}
> } catch (InterruptedIOException e) {
> t.interrupt();
> throw new ManifoldCFException("Interrupted: " + e.getMessage(),
> e, ManifoldCFException.INTERRUPTED);
> } catch (IOException e) {
> handleIOException(e);
> + } catch (ManifoldCFException e) {
> + handleManifoldCFException(e);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1730) Improvement suggestion for retry function in SharedDriveConnector
[
https://issues.apache.org/jira/browse/CONNECTORS-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601664#comment-17601664
]
Karl Wright commented on CONNECTORS-1730:
-
Hi, when an enhancement request is made it's important to understand why the
enhancement is needed. Are there situations you have encountered where the
numbers that are hardcoded are a problem? Or do you just not like hardcoded
numbers?
We have many cases of hardwired retry behavior in all of our connectors. See
all places where ServiceInterruption exception is thrown, for example. These
are generally modified only by developers because it would be far too complex
to support these rules in either the UI or in configuration information. The
complaints are not about the retry parameters, but usually around how we handle
very specific kinds of errors. There are many requests to "improve" connectors
by making them completely ignore errors, for example - and these are generally
not well thought out.
The burden is on you to demonstrate why these numbers need to be configurable.
> Improvement suggestion for retry function in SharedDriveConnector
> -
>
> Key: CONNECTORS-1730
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1730
> Project: ManifoldCF
> Issue Type: Improvement
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
>
> Hi there,
> I would like to suggest the following retry-related improvements:
> h3. +*1. Connector name*+
> SharedDriveConnector
> h3. +*2. Overview*+
> * When connection to SMB can't be executed, JCIFS connector will fail to
> connect and exception will occur. JCIFS will attempt to retry, and abort
> after a certain number of time.
> * The number of retry is currently controlled by the following parameters:
> ** *retriesRemaining* (hardcode：3)： The number of occurence of the same
> exception for a file or method. If a different exception occurs, this value
> is reset to 3.
> ** *totalTries* (hardcode：5)： The total number of occurrence of an exception
> for a file or method.
> For the two variables above, if *retriesRemaining* becomes 0 or *totalTries*
> becomes 5 then the job will be aborted.
> h3. +*3. Reasons for improvement*+
> Currently the maximum number of retry is being hardcoded at 3 and 5,
> respectively.
> In case connection to file server is unstable, to avoid aborting, I would
> like to suggest making these values customizable.
> For implementation, I would like to suggest the following methods:
> * 1/ Setting retry values in *properties.xml*
> * 2/ Setting retry values on WebUI of repository connection
> Between the two methods above, I suggest the first method because of
> following reasons:
> * The first method is easier to implement
> * Although the second method is more user-friendly, there are several issues:
> ** The config data from the screen will have to be stored in the database
> (PostgreSQL), resulting in an increased number of fields.
> ** Consequently, there might be a need to perform DB Migration in case
> further changes to setting field are needed.
> ※ According to the above reasons, I will proceed with the first method
> 'Setting retry values in properties.xml' for the next part of this suggestion.
> h3. +*4. Improvement*+
> Changing source code to read maximum number of retries from *properties.xml*
> Declare two variables in *properties.xml* to set the maximum number of retry:
> *
> `org.apache.manifoldcf.crawler.connectors.sharedrive.consecutivesmbexceptionretrycount`
> ⇒ Set to `consecutiveSMBExceptionRetryCount`
> * `org.apache.manifoldcf.crawler.connectors.sharedrive.totalsmbretrycount`
> ⇒ Set to `totalSMBRetryCount`
> E.g:
> {code:xml}
>name="org.apache.manifoldcf.crawler.connectors.sharedrive.consecutivesmbexceptionretrycount"
> value="3"/>
>name="org.apache.manifoldcf.crawler.connectors.sharedrive.totalsmbretrycount"
> value="5"/>
> {code}
> SharedDriveConnector will load these values from the file and set to two
> variables within the source code.
> ※In case these values can't be found from the file or set to an invalid
> value, default values will be used instead.
> h3. +*5. Suggested source code (based on release 2.22.1)*+
> Target class:
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector
> * Declare two class variables to store the configured values as follows:
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/jcifs/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/sharedrive/SharedDriveConnector.java#L103]
>
> {code:java}
> private final static int consecutiveSMBExceptionRetryCount;
> private final static int totalSMBRetryCount;
> {code}
>
> * Initialize the two variables above with following
[jira] [Assigned] (CONNECTORS-1730) Improvement suggestion for retry function in SharedDriveConnector
[
https://issues.apache.org/jira/browse/CONNECTORS-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1730:
---
Assignee: Karl Wright
> Improvement suggestion for retry function in SharedDriveConnector
> -
>
> Key: CONNECTORS-1730
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1730
> Project: ManifoldCF
> Issue Type: Improvement
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
>
> Hi there,
> I would like to suggest the following retry-related improvements:
> h3. +*1. Connector name*+
> SharedDriveConnector
> h3. +*2. Overview*+
> * When connection to SMB can't be executed, JCIFS connector will fail to
> connect and exception will occur. JCIFS will attempt to retry, and abort
> after a certain number of time.
> * The number of retry is currently controlled by the following parameters:
> ** *retriesRemaining* (hardcode：3)： The number of occurence of the same
> exception for a file or method. If a different exception occurs, this value
> is reset to 3.
> ** *totalTries* (hardcode：5)： The total number of occurrence of an exception
> for a file or method.
> For the two variables above, if *retriesRemaining* becomes 0 or *totalTries*
> becomes 5 then the job will be aborted.
> h3. +*3. Reasons for improvement*+
> Currently the maximum number of retry is being hardcoded at 3 and 5,
> respectively.
> In case connection to file server is unstable, to avoid aborting, I would
> like to suggest making these values customizable.
> For implementation, I would like to suggest the following methods:
> * 1/ Setting retry values in *properties.xml*
> * 2/ Setting retry values on WebUI of repository connection
> Between the two methods above, I suggest the first method because of
> following reasons:
> * The first method is easier to implement
> * Although the second method is more user-friendly, there are several issues:
> ** The config data from the screen will have to be stored in the database
> (PostgreSQL), resulting in an increased number of fields.
> ** Consequently, there might be a need to perform DB Migration in case
> further changes to setting field are needed.
> ※ According to the above reasons, I will proceed with the first method
> 'Setting retry values in properties.xml' for the next part of this suggestion.
> h3. +*4. Improvement*+
> Changing source code to read maximum number of retries from *properties.xml*
> Declare two variables in *properties.xml* to set the maximum number of retry:
> *
> `org.apache.manifoldcf.crawler.connectors.sharedrive.consecutivesmbexceptionretrycount`
> ⇒ Set to `consecutiveSMBExceptionRetryCount`
> * `org.apache.manifoldcf.crawler.connectors.sharedrive.totalsmbretrycount`
> ⇒ Set to `totalSMBRetryCount`
> E.g:
> {code:xml}
>name="org.apache.manifoldcf.crawler.connectors.sharedrive.consecutivesmbexceptionretrycount"
> value="3"/>
>name="org.apache.manifoldcf.crawler.connectors.sharedrive.totalsmbretrycount"
> value="5"/>
> {code}
> SharedDriveConnector will load these values from the file and set to two
> variables within the source code.
> ※In case these values can't be found from the file or set to an invalid
> value, default values will be used instead.
> h3. +*5. Suggested source code (based on release 2.22.1)*+
> Target class:
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector
> * Declare two class variables to store the configured values as follows:
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/jcifs/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/sharedrive/SharedDriveConnector.java#L103]
>
> {code:java}
> private final static int consecutiveSMBExceptionRetryCount;
> private final static int totalSMBRetryCount;
> {code}
>
> * Initialize the two variables above with following steps:
> ** Set the values configured in 'properties.xml' to the two variables above
> ** If these values weren't configured or invalid, set them to default values
> of 3 and 5, respectively.
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/jcifs/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/sharedrive/SharedDriveConnector.java#L106]
> {code:java}
> // Static initialization of various system properties. This hopefully
> takes place
> // before jcifs is loaded.
> static
> {
> ...
> int tempConsecutiveSMBExceptionRetryCount = 3;
> int tempTotalSMBRetryCount = 5;
> try {
> tempConsecutiveSMBExceptionRetryCount =
> ManifoldCF.getIntProperty("org.apache.manifoldcf.crawler.connectors.sharedrive.consecutivesmbexceptionretrycount",
> tempConsecutiveSMBExceptionRetryCount);
> } catch (ManifoldCFException e) {
>
[jira] [Resolved] (CONNECTORS-1729) The Confluence-v6 Repository Connector's attachment logic is incorrect
[
https://issues.apache.org/jira/browse/CONNECTORS-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1729.
-
Fix Version/s: ManifoldCF next
Resolution: Fixed
r1903771
> The Confluence-v6 Repository Connector's attachment logic is incorrect
> --
>
> Key: CONNECTORS-1729
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1729
> Project: ManifoldCF
> Issue Type: Bug
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF next
>
>
> Hi there,
> As there is an issue that is still not handled occurs in use, I would like to
> suggest the following fix for the source code of Confluence Repository
> Connector.
> For details about this issue, please refer to the information below:
> h3. +*1. Connector Name*+
> confluence-v6 \ Confluence Repository Connector
> h3. +*2. Overview*+
> * In the Confluence Repository Connector, there is an error in the logic
> that determines wether the document has attachments or not.
> * Wrong logic leads to attachments not being crawled.
> ※ This error only occurs when crawling documents from Confluence server,
> while crawling documents from Confluence Cloud (SaaS) still works normally.
> * Formats of the document's ID when there is a file attached are as below:
> ** Crawled from Confluence server: *-*
> ** Crawled from Confluence cloud (SaaS): *att- blog/page>*
> h3. +*3. Reproduction*+
> * On Confluence server:
> ** Create a blog.
> ** Add attachments to the newly created blog.
> * On ManifoldCF:
> ** Create a Confluence Repository Connector with the aforementioned
> Confluence server information.
> ** Create a job using the connector created above with the following details:
> *** On the [Page] tab:
> Process Attachments: (Check).
> Type Specification: Blog.
> ** Start job.
> ** Check [Simple History Report].
> h3. +*4. Cause*+
> * At the logic for judging whether the document has / does not have a file
> attachment, if the ID of the document begins with *att*, it is judging that
> there is a file attachment.
> * However, the ID field of the document crawled from the Confluence server,
> in fact, when the file is attached, does not prefix it with *att* (format
> mentioned in item 2).
> h3. +*5. Solution*+
> My observation is as below:
> * If a document has a file attachment, the ID of that document is a string
> of characters connected by *-* character.
> * If a document does not have a file attachment, the ID of that document
> does not contain *-* character.
> Therefore, it is possible to judge whether a file is is attached or not by
> checking if the ID contains *-* character.
> h3. +*6. Suggested source code (based on release 2.22.1)*+
> ***Class:
> org.apache.manifoldcf.crawler.connectors.confluence.v6.util.ConfluenceUtil***
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/confluence-v6/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/confluence/v6/util/ConfluenceUtil.java#L28]
> {code:java}
> - private static final String ATTACHMENT_ID_PREFIX = "att";
> + private static final String ATTACHMENT_ID_CHARACTER = "-";
> {code}
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/confluence-v6/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/confluence/v6/util/ConfluenceUtil.java#L47]
> {code:java}
>public static Boolean isAttachment(String id) {
> -return id.startsWith(ATTACHMENT_ID_PREFIX);
> +return id.contains(ATTACHMENT_ID_CHARACTER);
>}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CONNECTORS-1729) The Confluence-v6 Repository Connector's attachment logic is incorrect
[
https://issues.apache.org/jira/browse/CONNECTORS-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1729:
---
Assignee: Karl Wright
> The Confluence-v6 Repository Connector's attachment logic is incorrect
> --
>
> Key: CONNECTORS-1729
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1729
> Project: ManifoldCF
> Issue Type: Bug
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
>
> Hi there,
> As there is an issue that is still not handled occurs in use, I would like to
> suggest the following fix for the source code of Confluence Repository
> Connector.
> For details about this issue, please refer to the information below:
> h3. +*1. Connector Name*+
> confluence-v6 \ Confluence Repository Connector
> h3. +*2. Overview*+
> * In the Confluence Repository Connector, there is an error in the logic
> that determines wether the document has attachments or not.
> * Wrong logic leads to attachments not being crawled.
> ※ This error only occurs when crawling documents from Confluence server,
> while crawling documents from Confluence Cloud (SaaS) still works normally.
> * Formats of the document's ID when there is a file attached are as below:
> ** Crawled from Confluence server: *-*
> ** Crawled from Confluence cloud (SaaS): *att- blog/page>*
> h3. +*3. Reproduction*+
> * On Confluence server:
> ** Create a blog.
> ** Add attachments to the newly created blog.
> * On ManifoldCF:
> ** Create a Confluence Repository Connector with the aforementioned
> Confluence server information.
> ** Create a job using the connector created above with the following details:
> *** On the [Page] tab:
> Process Attachments: (Check).
> Type Specification: Blog.
> ** Start job.
> ** Check [Simple History Report].
> h3. +*4. Cause*+
> * At the logic for judging whether the document has / does not have a file
> attachment, if the ID of the document begins with *att*, it is judging that
> there is a file attachment.
> * However, the ID field of the document crawled from the Confluence server,
> in fact, when the file is attached, does not prefix it with *att* (format
> mentioned in item 2).
> h3. +*5. Solution*+
> My observation is as below:
> * If a document has a file attachment, the ID of that document is a string
> of characters connected by *-* character.
> * If a document does not have a file attachment, the ID of that document
> does not contain *-* character.
> Therefore, it is possible to judge whether a file is is attached or not by
> checking if the ID contains *-* character.
> h3. +*6. Suggested source code (based on release 2.22.1)*+
> ***Class:
> org.apache.manifoldcf.crawler.connectors.confluence.v6.util.ConfluenceUtil***
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/confluence-v6/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/confluence/v6/util/ConfluenceUtil.java#L28]
> {code:java}
> - private static final String ATTACHMENT_ID_PREFIX = "att";
> + private static final String ATTACHMENT_ID_CHARACTER = "-";
> {code}
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/confluence-v6/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/confluence/v6/util/ConfluenceUtil.java#L47]
> {code:java}
>public static Boolean isAttachment(String id) {
> -return id.startsWith(ATTACHMENT_ID_PREFIX);
> +return id.contains(ATTACHMENT_ID_CHARACTER);
>}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CONNECTORS-1727) Timeout values for Genreric Authority is not updated after setting
[
https://issues.apache.org/jira/browse/CONNECTORS-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1727.
-
Fix Version/s: ManifoldCF next
Resolution: Fixed
r1903770.
> Timeout values for Genreric Authority is not updated after setting
> --
>
> Key: CONNECTORS-1727
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1727
> Project: ManifoldCF
> Issue Type: Bug
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF next
>
>
> Hi there,
> As there is a problem that is still not addressed during use, I would like to
> suggest the following correction for the source code of the Generic Authority
> Connector.
> ※This is the same issue as that in Generic Repository Connector, which was
> resolved at
> [CONNECTORS-1726|https://issues.apache.org/jira/browse/CONNECTORS-1726]
> For additional details, please see below:
> h3. +*1. Connector name*+
> Generic Authority Connector
> h3. +*2. Issue*+
> When I create or edit a Generic authority connection, I cannot update the
> value in the following fields:
> * Connection timeout (milis)
> * Socket timeout (milis)
> h3. +*3. Reproduction*+
> * Create a Generic authority connection
> ** On *Entry point* tab, edit the values of *Connection timeout (milis)* and
> *Socket timeout (milis)* fields
> ** Click on *Save* button
> * On *View Authority Connection Status - Generic* screen, it can be seen
> that the values of the 2 above fields are not updated.
> h3. +*4. Cause*+
> The names of the textboxes for the 2 fields are the followings:
> * genericConTimeout
> * genericSoTimeout
> However, the names that are being used inside the source code are the
> followings:
> * genericConnectionTimeout
> * genericSocketTimeout
> This results in that new values can not be obtained, thus the values of the 2
> fields can not be updated.
> h3. +*5. Solution*+
> Update parameter names for Connection Timeout and Socket Timeout with names
> that are being stored inside the DataBase:
> * genericConTimeout ➞ genericConnectionTimeout
> * genericSoTimeout ➞ genericSocketTimeout
> h3. +*6. Suggested source code (based on release 2.22.1)*+
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/generic/GenericAuthority.java#L400]
> {code:java}
> + " \n"
> + " " +
> Messages.getBodyString(locale, "generic.ConnectionTimeoutColon") +
> "\n"
> - + " name=\"genericConTimeout\" value=\"" + Encoder.attributeEscape(conTimeout) +
> "\"/>\n"
> + + " name=\"genericConnectionTimeout\" value=\"" +
> Encoder.attributeEscape(conTimeout) + "\"/>\n"
> + " \n"
> + " \n"
> + " " +
> Messages.getBodyString(locale, "generic.SocketTimeoutColon") +
> "\n"
> - + " name=\"genericSoTimeout\" value=\"" + Encoder.attributeEscape(soTimeout) +
> "\"/>\n"
> + + " name=\"genericSocketTimeout\" value=\"" + Encoder.attributeEscape(soTimeout)
> + "\"/>\n"
> + " \n"
> {code}
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/generic/GenericAuthority.java#L415]
> {code:java}
> - out.print(" + Encoder.attributeEscape(conTimeout) + "\"/>\n");
> - out.print(" Encoder.attributeEscape(soTimeout) + "\"/>\n");
> + out.print(" value=\"" + Encoder.attributeEscape(conTimeout) + "\"/>\n");
> + out.print(" value=\"" + Encoder.attributeEscape(soTimeout) + "\"/>\n");
> {code}
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/generic/GenericAuthority.java#L428]
> {code:java}
> - copyParam(variableContext, parameters, "genericConTimeout");
> - copyParam(variableContext, parameters, "genericSoTimeout");
> + copyParam(variableContext, parameters, "genericConnectionTimeout");
> + copyParam(variableContext, parameters, "genericSocketTimeout");
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CONNECTORS-1728) Fix error message of Generic Repository Connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1728:
---
Assignee: Karl Wright
> Fix error message of Generic Repository Connector
> -
>
> Key: CONNECTORS-1728
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1728
> Project: ManifoldCF
> Issue Type: Bug
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Minor
>
> Hi there,
> As there is a problem that is still not addressed during use, I would like to
> suggest the following correction for the source code of the Generic
> Repository Connector.
> For additional details, please see below:
> h3. +*1. Connector name*+
> Generic Repository Connector
> h3. +*2. Issue*+
> In the *run()* method of the *GenericConnector$DocumentVersionThread* class,
> if connector cannot connect to REST API (HTTP status code != 200), there is
> an error message in log file:
> [ *addSeedDocuments error* - interface returned incorrect return code for:
> ... ]
> However, this is *DocumentVersionThread* thread, not *ExecuteSeedingThread*
> thread. The *addSeedDocuments error* prefix is not suiable to this thread.
> I think it should be *getDocumentVersions error* prefix.
> h3. +*3. Cause*+
> This may be a copy/paste mistake:
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L1207]
> {code:java}
> if (response.getStatusLine().getStatusCode() != HttpStatus.SC_OK) {
> exception = new ManifoldCFException("addSeedDocuments error -
> interface returned incorrect return code for: " + url + " - " +
> response.getStatusLine().toString());
> return;
> }
> {code}
> h3. +*4. Solution*+
> Updating the content of this error message from [addSeedDocuments error] to
> [getDocumentVersions error]
> h3. +*5. Suggested source code (based on release 2.22.1)*+
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L1207]
> {code:java}
> if (response.getStatusLine().getStatusCode() != HttpStatus.SC_OK) {
> - exception = new ManifoldCFException("addSeedDocuments error -
> interface returned incorrect return code for: " + url + " - " +
> response.getStatusLine().toString());
> + exception = new ManifoldCFException("getDocumentVersions error -
> interface returned incorrect return code for: " + url + " - " +
> response.getStatusLine().toString());
> return;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CONNECTORS-1728) Fix error message of Generic Repository Connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1728.
-
Fix Version/s: ManifoldCF next
Resolution: Fixed
r1903770
> Fix error message of Generic Repository Connector
> -
>
> Key: CONNECTORS-1728
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1728
> Project: ManifoldCF
> Issue Type: Bug
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Minor
> Fix For: ManifoldCF next
>
>
> Hi there,
> As there is a problem that is still not addressed during use, I would like to
> suggest the following correction for the source code of the Generic
> Repository Connector.
> For additional details, please see below:
> h3. +*1. Connector name*+
> Generic Repository Connector
> h3. +*2. Issue*+
> In the *run()* method of the *GenericConnector$DocumentVersionThread* class,
> if connector cannot connect to REST API (HTTP status code != 200), there is
> an error message in log file:
> [ *addSeedDocuments error* - interface returned incorrect return code for:
> ... ]
> However, this is *DocumentVersionThread* thread, not *ExecuteSeedingThread*
> thread. The *addSeedDocuments error* prefix is not suiable to this thread.
> I think it should be *getDocumentVersions error* prefix.
> h3. +*3. Cause*+
> This may be a copy/paste mistake:
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L1207]
> {code:java}
> if (response.getStatusLine().getStatusCode() != HttpStatus.SC_OK) {
> exception = new ManifoldCFException("addSeedDocuments error -
> interface returned incorrect return code for: " + url + " - " +
> response.getStatusLine().toString());
> return;
> }
> {code}
> h3. +*4. Solution*+
> Updating the content of this error message from [addSeedDocuments error] to
> [getDocumentVersions error]
> h3. +*5. Suggested source code (based on release 2.22.1)*+
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L1207]
> {code:java}
> if (response.getStatusLine().getStatusCode() != HttpStatus.SC_OK) {
> - exception = new ManifoldCFException("addSeedDocuments error -
> interface returned incorrect return code for: " + url + " - " +
> response.getStatusLine().toString());
> + exception = new ManifoldCFException("getDocumentVersions error -
> interface returned incorrect return code for: " + url + " - " +
> response.getStatusLine().toString());
> return;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CONNECTORS-1727) Timeout values for Genreric Authority is not updated after setting
[
https://issues.apache.org/jira/browse/CONNECTORS-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1727:
---
Assignee: Karl Wright
> Timeout values for Genreric Authority is not updated after setting
> --
>
> Key: CONNECTORS-1727
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1727
> Project: ManifoldCF
> Issue Type: Bug
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
>
> Hi there,
> As there is a problem that is still not addressed during use, I would like to
> suggest the following correction for the source code of the Generic Authority
> Connector.
> ※This is the same issue as that in Generic Repository Connector, which was
> resolved at
> [CONNECTORS-1726|https://issues.apache.org/jira/browse/CONNECTORS-1726]
> For additional details, please see below:
> h3. +*1. Connector name*+
> Generic Authority Connector
> h3. +*2. Issue*+
> When I create or edit a Generic authority connection, I cannot update the
> value in the following fields:
> * Connection timeout (milis)
> * Socket timeout (milis)
> h3. +*3. Reproduction*+
> * Create a Generic authority connection
> ** On *Entry point* tab, edit the values of *Connection timeout (milis)* and
> *Socket timeout (milis)* fields
> ** Click on *Save* button
> * On *View Authority Connection Status - Generic* screen, it can be seen
> that the values of the 2 above fields are not updated.
> h3. +*4. Cause*+
> The names of the textboxes for the 2 fields are the followings:
> * genericConTimeout
> * genericSoTimeout
> However, the names that are being used inside the source code are the
> followings:
> * genericConnectionTimeout
> * genericSocketTimeout
> This results in that new values can not be obtained, thus the values of the 2
> fields can not be updated.
> h3. +*5. Solution*+
> Update parameter names for Connection Timeout and Socket Timeout with names
> that are being stored inside the DataBase:
> * genericConTimeout ➞ genericConnectionTimeout
> * genericSoTimeout ➞ genericSocketTimeout
> h3. +*6. Suggested source code (based on release 2.22.1)*+
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/generic/GenericAuthority.java#L400]
> {code:java}
> + " \n"
> + " " +
> Messages.getBodyString(locale, "generic.ConnectionTimeoutColon") +
> "\n"
> - + " name=\"genericConTimeout\" value=\"" + Encoder.attributeEscape(conTimeout) +
> "\"/>\n"
> + + " name=\"genericConnectionTimeout\" value=\"" +
> Encoder.attributeEscape(conTimeout) + "\"/>\n"
> + " \n"
> + " \n"
> + " " +
> Messages.getBodyString(locale, "generic.SocketTimeoutColon") +
> "\n"
> - + " name=\"genericSoTimeout\" value=\"" + Encoder.attributeEscape(soTimeout) +
> "\"/>\n"
> + + " name=\"genericSocketTimeout\" value=\"" + Encoder.attributeEscape(soTimeout)
> + "\"/>\n"
> + " \n"
> {code}
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/generic/GenericAuthority.java#L415]
> {code:java}
> - out.print(" + Encoder.attributeEscape(conTimeout) + "\"/>\n");
> - out.print(" Encoder.attributeEscape(soTimeout) + "\"/>\n");
> + out.print(" value=\"" + Encoder.attributeEscape(conTimeout) + "\"/>\n");
> + out.print(" value=\"" + Encoder.attributeEscape(soTimeout) + "\"/>\n");
> {code}
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/authorities/authorities/generic/GenericAuthority.java#L428]
> {code:java}
> - copyParam(variableContext, parameters, "genericConTimeout");
> - copyParam(variableContext, parameters, "genericSoTimeout");
> + copyParam(variableContext, parameters, "genericConnectionTimeout");
> + copyParam(variableContext, parameters, "genericSocketTimeout");
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CONNECTORS-1726) Timeout values for Genreric Repository is not updated after setting
[
https://issues.apache.org/jira/browse/CONNECTORS-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1726.
-
Fix Version/s: ManifoldCF next
Resolution: Fixed
r1903752
> Timeout values for Genreric Repository is not updated after setting
> ---
>
> Key: CONNECTORS-1726
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1726
> Project: ManifoldCF
> Issue Type: Bug
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF next
>
>
> Hi there,
> As there is a problem that is still not addressed during use, I would like to
> suggest the following correction for the source code of the Generic
> Repository Connector.
> For additional details, please see below:
> h3. +*1. Connector name*+
> Generic Repository Connector
> h3. +*2. Issue*+
> When I create or edit a Generic repository connection, I cannot update the
> value in the following fields:
> * Connection timeout (milis)
> * Socket timeout (milis)
> h3. +*3. Reproduction*+
> * Create a Generic repository connection
> ** On *Entry point* tab, edit the values of *Connection timeout (milis)* and
> *Socket timeout (milis)* fields
> ** Click on *Save* button
> * On *View Repository Connection Status - Generic* screen, it can be seen
> that the values of the 2 above fields are not updated.
> h3. +*4. Cause*+
> The names of the textboxes for the 2 fields are the followings:
> * genericConTimeout
> * genericSoTimeout
> However, the names that are being used inside the source code are the
> followings:
> * genericConnectionTimeout
> * genericSocketTimeout
> This results in that new values can not be obtained, thus the values of the 2
> fields can not be updated.
> h3. +*5. Solution*+
> Update parameter names for Connection Timeout and Socket Timeout with names
> that are being stored inside the DataBase:
> * genericConTimeout ➞ genericConnectionTimeout
> * genericSoTimeout ➞ genericSocketTimeout
> h3. +*6. Suggested source code (based on release 2.22.1)*+
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L510]
> {code:java}
> + " \n"
> + " " +
> Messages.getBodyString(locale, "generic.ConnectionTimeoutColon") +
> "\n"
> - + " name=\"genericConTimeout\" value=\"" + Encoder.attributeEscape(conTimeout) +
> "\"/>\n"
> + + " name=\"genericConnectionTimeout\" value=\"" +
> Encoder.attributeEscape(conTimeout) + "\"/>\n"
> + " \n"
> + " \n"
> + " " +
> Messages.getBodyString(locale, "generic.SocketTimeoutColon") +
> "\n"
> - + " name=\"genericSoTimeout\" value=\"" + Encoder.attributeEscape(soTimeout) +
> "\"/>\n"
> + + " name=\"genericSocketTimeout\" value=\"" + Encoder.attributeEscape(soTimeout)
> + "\"/>\n"
> + " \n"
> {code}
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L523]
> {code:java}
> - out.print(" + Encoder.attributeEscape(conTimeout) + "\"/>\n");
> - out.print(" Encoder.attributeEscape(soTimeout) + "\"/>\n");
> + out.print(" value=\"" + Encoder.attributeEscape(conTimeout) + "\"/>\n");
> + out.print(" value=\"" + Encoder.attributeEscape(soTimeout) + "\"/>\n");
> {code}
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L535]
> {code:java}
> - copyParam(variableContext, parameters, "genericConTimeout");
> - copyParam(variableContext, parameters, "genericSoTimeout");
> + copyParam(variableContext, parameters, "genericConnectionTimeout");
> + copyParam(variableContext, parameters, "genericSocketTimeout");
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CONNECTORS-1726) Timeout values for Genreric Repository is not updated after setting
[
https://issues.apache.org/jira/browse/CONNECTORS-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1726:
---
Assignee: Karl Wright
> Timeout values for Genreric Repository is not updated after setting
> ---
>
> Key: CONNECTORS-1726
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1726
> Project: ManifoldCF
> Issue Type: Bug
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
>
> Hi there,
> As there is a problem that is still not addressed during use, I would like to
> suggest the following correction for the source code of the Generic
> Repository Connector.
> For additional details, please see below:
> h3. +*1. Connector name*+
> Generic Repository Connector
> h3. +*2. Issue*+
> When I create or edit a Generic repository connection, I cannot update the
> value in the following fields:
> * Connection timeout (milis)
> * Socket timeout (milis)
> h3. +*3. Reproduction*+
> * Create a Generic repository connection
> ** On *Entry point* tab, edit the values of *Connection timeout (milis)* and
> *Socket timeout (milis)* fields
> ** Click on *Save* button
> * On *View Repository Connection Status - Generic* screen, it can be seen
> that the values of the 2 above fields are not updated.
> h3. +*4. Cause*+
> The names of the textboxes for the 2 fields are the followings:
> * genericConTimeout
> * genericSoTimeout
> However, the names that are being used inside the source code are the
> followings:
> * genericConnectionTimeout
> * genericSocketTimeout
> This results in that new values can not be obtained, thus the values of the 2
> fields can not be updated.
> h3. +*5. Solution*+
> Update parameter names for Connection Timeout and Socket Timeout with names
> that are being stored inside the DataBase:
> * genericConTimeout ➞ genericConnectionTimeout
> * genericSoTimeout ➞ genericSocketTimeout
> h3. +*6. Suggested source code (based on release 2.22.1)*+
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L510]
> {code:java}
> + " \n"
> + " " +
> Messages.getBodyString(locale, "generic.ConnectionTimeoutColon") +
> "\n"
> - + " name=\"genericConTimeout\" value=\"" + Encoder.attributeEscape(conTimeout) +
> "\"/>\n"
> + + " name=\"genericConnectionTimeout\" value=\"" +
> Encoder.attributeEscape(conTimeout) + "\"/>\n"
> + " \n"
> + " \n"
> + " " +
> Messages.getBodyString(locale, "generic.SocketTimeoutColon") +
> "\n"
> - + " name=\"genericSoTimeout\" value=\"" + Encoder.attributeEscape(soTimeout) +
> "\"/>\n"
> + + " name=\"genericSocketTimeout\" value=\"" + Encoder.attributeEscape(soTimeout)
> + "\"/>\n"
> + " \n"
> {code}
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L523]
> {code:java}
> - out.print(" + Encoder.attributeEscape(conTimeout) + "\"/>\n");
> - out.print(" Encoder.attributeEscape(soTimeout) + "\"/>\n");
> + out.print(" value=\"" + Encoder.attributeEscape(conTimeout) + "\"/>\n");
> + out.print(" value=\"" + Encoder.attributeEscape(soTimeout) + "\"/>\n");
> {code}
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L535]
> {code:java}
> - copyParam(variableContext, parameters, "genericConTimeout");
> - copyParam(variableContext, parameters, "genericSoTimeout");
> + copyParam(variableContext, parameters, "genericConnectionTimeout");
> + copyParam(variableContext, parameters, "genericSocketTimeout");
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CONNECTORS-1725) MissingResourceException exception occurs at Generic Repository Connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1725:
---
Assignee: Karl Wright
> MissingResourceException exception occurs at Generic Repository Connector
> -
>
> Key: CONNECTORS-1725
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1725
> Project: ManifoldCF
> Issue Type: Bug
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Minor
>
> Hi there,
> As there is a problem that is still not addressed during use, I would like to
> suggest the following correction for the source code of the Generic
> Repository Connector.
> For additional details, please see below:
> h3. +*1. Connector name*+
> Generic Repository Connector
> h3. +*2. Issue*+
> When I create or edit a job using Generic Repository Connector, if I add a
> parameter without designate its *Parameter name* field, the alert message
> *generic.TypeInParamName* appears.
> An error log is as follows:
> {noformat}
> ERROR 2022-08-04T15:45:43,443 (qtp10405169-442) - Missing resource
> 'generic.TypeInParamName' in bundle
> 'org.apache.manifoldcf.crawler.connectors.generic.common' for locale 'en'
> java.util.MissingResourceException: Can't find resource for bundle
> java.util.PropertyResourceBundle, key generic.TypeInParamName
> at java.util.ResourceBundle.getObject(ResourceBundle.java:450)
> ~[?:1.8.0_211]
> at java.util.ResourceBundle.getString(ResourceBundle.java:407)
> ~[?:1.8.0_211]
> at org.apache.manifoldcf.core.i18n.Messages.getMessage(Messages.java:195)
> ~[mcf-core.jar:?]
> at org.apache.manifoldcf.core.i18n.Messages.getMessage(Messages.java:184)
> ~[mcf-core.jar:?]
> at org.apache.manifoldcf.core.i18n.Messages.getString(Messages.java:218)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.getBodyJavascriptString(Messages.java:343)
> ~[mcf-ui-core.jar:?]
> at
> org.apache.manifoldcf.crawler.connectors.generic.Messages.getBodyJavascriptString(Messages.java:95)
> ~[?:?]
> at
> org.apache.manifoldcf.crawler.connectors.generic.Messages.getBodyJavascriptString(Messages.java:54)
> ~[?:?]
> at
> org.apache.manifoldcf.crawler.connectors.generic.GenericConnector.outputSpecificationHeader(GenericConnector.java:610)
> ~[?:?]
> {noformat}
> h3. +*3. Reproduction*+
> * Create a Generic Repository Connector
> * Create a job using the connector created above with the following details:
> ** On the Parameters tab, add the following parameters:
> *** Parameter name: blank
> *** Parameter value:
> *** Click [Add]
> h3. +*4. Cause*+
> Key *generic.TypeInParamName* is not present in native2ascii _*.properties_
> files, however it is in use.
> Perhaps, this key is being mistaken with *generic.TypeInParameterName*, that
> is not in use.
> h3. +*5. Solution*+
> Update the property key used in java classes so that it matches the one
> defined in _*.properties_ files:
> *generic.TypeInParamName* ➞ *generic.TypeInParameterName*
> h3. +*6. Suggested source code (based on release 2.22.1)*+
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L610]
> {code:java}
> + "function "+seqPrefix+"SpecAddParam(anchorvalue) {\n"
> + " if (editjob."+seqPrefix+"specparamname.value == \"\")\n"
> + " {\n"
> - + "alert(\"" + Messages.getBodyJavascriptString(locale,
> "generic.TypeInParamName") + "\");\n"
> + + "alert(\"" + Messages.getBodyJavascriptString(locale,
> "generic.TypeInParameterName") + "\");\n"
> + "editjob."+seqPrefix+"specparamname.focus();\n"
> + "return;\n"
> + " }\n"
> + "
> "+seqPrefix+"SpecOp(\""+seqPrefix+"paramop\",\"Add\",anchorvalue);\n"
> + "}\n"
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CONNECTORS-1725) MissingResourceException exception occurs at Generic Repository Connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1725.
-
Fix Version/s: ManifoldCF next
Resolution: Fixed
r1903752
> MissingResourceException exception occurs at Generic Repository Connector
> -
>
> Key: CONNECTORS-1725
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1725
> Project: ManifoldCF
> Issue Type: Bug
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Minor
> Fix For: ManifoldCF next
>
>
> Hi there,
> As there is a problem that is still not addressed during use, I would like to
> suggest the following correction for the source code of the Generic
> Repository Connector.
> For additional details, please see below:
> h3. +*1. Connector name*+
> Generic Repository Connector
> h3. +*2. Issue*+
> When I create or edit a job using Generic Repository Connector, if I add a
> parameter without designate its *Parameter name* field, the alert message
> *generic.TypeInParamName* appears.
> An error log is as follows:
> {noformat}
> ERROR 2022-08-04T15:45:43,443 (qtp10405169-442) - Missing resource
> 'generic.TypeInParamName' in bundle
> 'org.apache.manifoldcf.crawler.connectors.generic.common' for locale 'en'
> java.util.MissingResourceException: Can't find resource for bundle
> java.util.PropertyResourceBundle, key generic.TypeInParamName
> at java.util.ResourceBundle.getObject(ResourceBundle.java:450)
> ~[?:1.8.0_211]
> at java.util.ResourceBundle.getString(ResourceBundle.java:407)
> ~[?:1.8.0_211]
> at org.apache.manifoldcf.core.i18n.Messages.getMessage(Messages.java:195)
> ~[mcf-core.jar:?]
> at org.apache.manifoldcf.core.i18n.Messages.getMessage(Messages.java:184)
> ~[mcf-core.jar:?]
> at org.apache.manifoldcf.core.i18n.Messages.getString(Messages.java:218)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.getBodyJavascriptString(Messages.java:343)
> ~[mcf-ui-core.jar:?]
> at
> org.apache.manifoldcf.crawler.connectors.generic.Messages.getBodyJavascriptString(Messages.java:95)
> ~[?:?]
> at
> org.apache.manifoldcf.crawler.connectors.generic.Messages.getBodyJavascriptString(Messages.java:54)
> ~[?:?]
> at
> org.apache.manifoldcf.crawler.connectors.generic.GenericConnector.outputSpecificationHeader(GenericConnector.java:610)
> ~[?:?]
> {noformat}
> h3. +*3. Reproduction*+
> * Create a Generic Repository Connector
> * Create a job using the connector created above with the following details:
> ** On the Parameters tab, add the following parameters:
> *** Parameter name: blank
> *** Parameter value:
> *** Click [Add]
> h3. +*4. Cause*+
> Key *generic.TypeInParamName* is not present in native2ascii _*.properties_
> files, however it is in use.
> Perhaps, this key is being mistaken with *generic.TypeInParameterName*, that
> is not in use.
> h3. +*5. Solution*+
> Update the property key used in java classes so that it matches the one
> defined in _*.properties_ files:
> *generic.TypeInParamName* ➞ *generic.TypeInParameterName*
> h3. +*6. Suggested source code (based on release 2.22.1)*+
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L610]
> {code:java}
> + "function "+seqPrefix+"SpecAddParam(anchorvalue) {\n"
> + " if (editjob."+seqPrefix+"specparamname.value == \"\")\n"
> + " {\n"
> - + "alert(\"" + Messages.getBodyJavascriptString(locale,
> "generic.TypeInParamName") + "\");\n"
> + + "alert(\"" + Messages.getBodyJavascriptString(locale,
> "generic.TypeInParameterName") + "\");\n"
> + "editjob."+seqPrefix+"specparamname.focus();\n"
> + "return;\n"
> + " }\n"
> + "
> "+seqPrefix+"SpecOp(\""+seqPrefix+"paramop\",\"Add\",anchorvalue);\n"
> + "}\n"
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CONNECTORS-1724) When the REST API cannot be connected, job using the Generic Repository Connector would be freezed.
[
https://issues.apache.org/jira/browse/CONNECTORS-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1724.
-
Fix Version/s: ManifoldCF next
Resolution: Fixed
r1903673
> When the REST API cannot be connected, job using the Generic Repository
> Connector would be freezed.
> ---
>
> Key: CONNECTORS-1724
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1724
> Project: ManifoldCF
> Issue Type: Bug
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF next
>
>
> Hi there,
> As there is an issue that is still not handled occurs in use, I would like to
> suggest the following fix for the source code of Generic repository connector.
> For details about this issue, please refer to the information below:
> h3. +*1. Connector name*+
> Generic Repository Connector
> h3. +*2. Issue*+
> When Generic Repository is calling REST API with _action=seed_ and an error
> occurs, corresponding error handling is not executed, which results in that
> crawling job of ManifoldCF is frozen at status *Starting up* and no error
> message is outputted.
> * When this issue happens in the Generic Repository, seed phase of jobs in
> other repositories also freezes (perhaps, seed thread is also frozen)
> * Even after ManifoldCF is restarted, as jobs are automatically executed,
> the same issue happens again.
> * A temporary solution is to aborting the job and recheck the connection.
> h3. +*3. Reproduction*+
> h4. *Reproduction method:*
> * At setting step for Generic repository connection, set a non-existent
> entry point (e.g. [http://localhost/no*exist/]). Then, define a job that uses
> that entry point and run that job.
> * 10 minutes or more after the job gets started, its status is still
> *Starting up* and abnormal end does not occur due to connection error and
> time-out.
> h4. *Reproduction steps:*
> * Create a Generic repository connection with the following settings:
> ** On the *Entry Point* tab, set a non-existent entry point （e.g.
> [http://localhost/no*exist/])
> * Create a job using above Generic repository connection
> * Start the created job and keep track of its status
> ** Job is going to be frozen with the following information:
> *** Status: Starting up
> *** Start Time: Not started
> *** Documents: 0
> ** No new events appear in *Document Status*
> ** No errors get logged in manifoldcf.log
> h3. +*4. Cause*+
> In *GenericConnector$ExecuteSeedingThread* class, *seedBuffer.signalDone()*
> method is only called when returned HTTP status code is 200.
> * When the connector is not able to connect to REST API, which means that
> returned HTTP status code is not 200, *seedBuffer.signalDone()* method is not
> called.
> ** This results in that *complete* flag is not reassigned as _true_
> ** As *complete* flag is not reassigned as _true_ and *buffer.size()* is 0,
> job is stuck in the *wait()* process, inside the while loop of
> *XThreadStringBuffer#fetch()* method.
> ([https://github.com/apache/manifoldcf/blob/release-2.22.1/framework/connector-common/src/main/java/org/apache/manifoldcf/connectorcommon/common/XThreadStringBuffer.java#L78])
> {code:java}
> while (buffer.size() == 0 && !complete)
> wait();
> {code}
> ⇒ These are the reasons why job is frozen at status *Starting up*
> h3. +*5. Solution*+
> In order to resolve this issue, we suggest the following things:
> * *seedBuffer.signalDone()* method should be called for all cases of HTTP
> response status.
> * Moreover, when HTTP status code is not 200, ManifoldCFException is thrown.
> There is no process to handle ManifoldCFException in *finishUp()* method of
> *GenericConnector$ExecuteSeedingThread* class, so process to handle this
> exception should be added.
> h3. +*6. Suggested source code (based on release 2.22.1)*+
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L1151]
> {code:java}
> - seedBuffer.signalDone();
> } finally {
> EntityUtils.consume(response.getEntity());
> method.releaseConnection();
> + seedBuffer.signalDone();
> }
> {code}
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L1120]
> {code:java}
> if (thr instanceof RuntimeException) {
> throw (RuntimeException) thr;
> } else if (thr instanceof Error) {
> throw (Error) thr;
> + } else if (thr instanceof ManifoldCFException) {
> + throw (ManifoldCFException)
[jira] [Assigned] (CONNECTORS-1724) When the REST API cannot be connected, job using the Generic Repository Connector would be freezed.
[
https://issues.apache.org/jira/browse/CONNECTORS-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1724:
---
Assignee: Karl Wright
> When the REST API cannot be connected, job using the Generic Repository
> Connector would be freezed.
> ---
>
> Key: CONNECTORS-1724
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1724
> Project: ManifoldCF
> Issue Type: Bug
>Reporter: Nguyen Huu Nhat
>Assignee: Karl Wright
>Priority: Major
>
> Hi there,
> As there is an issue that is still not handled occurs in use, I would like to
> suggest the following fix for the source code of Generic repository connector.
> For details about this issue, please refer to the information below:
> h3. +*1. Connector name*+
> Generic Repository Connector
> h3. +*2. Issue*+
> When Generic Repository is calling REST API with _action=seed_ and an error
> occurs, corresponding error handling is not executed, which results in that
> crawling job of ManifoldCF is frozen at status *Starting up* and no error
> message is outputted.
> * When this issue happens in the Generic Repository, seed phase of jobs in
> other repositories also freezes (perhaps, seed thread is also frozen)
> * Even after ManifoldCF is restarted, as jobs are automatically executed,
> the same issue happens again.
> * A temporary solution is to aborting the job and recheck the connection.
> h3. +*3. Reproduction*+
> h4. *Reproduction method:*
> * At setting step for Generic repository connection, set a non-existent
> entry point (e.g. [http://localhost/no*exist/]). Then, define a job that uses
> that entry point and run that job.
> * 10 minutes or more after the job gets started, its status is still
> *Starting up* and abnormal end does not occur due to connection error and
> time-out.
> h4. *Reproduction steps:*
> * Create a Generic repository connection with the following settings:
> ** On the *Entry Point* tab, set a non-existent entry point （e.g.
> [http://localhost/no*exist/])
> * Create a job using above Generic repository connection
> * Start the created job and keep track of its status
> ** Job is going to be frozen with the following information:
> *** Status: Starting up
> *** Start Time: Not started
> *** Documents: 0
> ** No new events appear in *Document Status*
> ** No errors get logged in manifoldcf.log
> h3. +*4. Cause*+
> In *GenericConnector$ExecuteSeedingThread* class, *seedBuffer.signalDone()*
> method is only called when returned HTTP status code is 200.
> * When the connector is not able to connect to REST API, which means that
> returned HTTP status code is not 200, *seedBuffer.signalDone()* method is not
> called.
> ** This results in that *complete* flag is not reassigned as _true_
> ** As *complete* flag is not reassigned as _true_ and *buffer.size()* is 0,
> job is stuck in the *wait()* process, inside the while loop of
> *XThreadStringBuffer#fetch()* method.
> ([https://github.com/apache/manifoldcf/blob/release-2.22.1/framework/connector-common/src/main/java/org/apache/manifoldcf/connectorcommon/common/XThreadStringBuffer.java#L78])
> {code:java}
> while (buffer.size() == 0 && !complete)
> wait();
> {code}
> ⇒ These are the reasons why job is frozen at status *Starting up*
> h3. +*5. Solution*+
> In order to resolve this issue, we suggest the following things:
> * *seedBuffer.signalDone()* method should be called for all cases of HTTP
> response status.
> * Moreover, when HTTP status code is not 200, ManifoldCFException is thrown.
> There is no process to handle ManifoldCFException in *finishUp()* method of
> *GenericConnector$ExecuteSeedingThread* class, so process to handle this
> exception should be added.
> h3. +*6. Suggested source code (based on release 2.22.1)*+
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L1151]
> {code:java}
> - seedBuffer.signalDone();
> } finally {
> EntityUtils.consume(response.getEntity());
> method.releaseConnection();
> + seedBuffer.signalDone();
> }
> {code}
> [https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L1120]
> {code:java}
> if (thr instanceof RuntimeException) {
> throw (RuntimeException) thr;
> } else if (thr instanceof Error) {
> throw (Error) thr;
> + } else if (thr instanceof ManifoldCFException) {
> + throw (ManifoldCFException) thr;
> } else {
> throw new RuntimeException("Unhandled
[jira] [Resolved] (CONNECTORS-1723) Need Solr 9.x plugin
[ https://issues.apache.org/jira/browse/CONNECTORS-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1723. - Resolution: Fixed > Need Solr 9.x plugin > > > Key: CONNECTORS-1723 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1723 > Project: ManifoldCF > Issue Type: New Feature >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Attachments: patch.txt > > > Please attach patch with code we will use for plugin. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CONNECTORS-1723) Need Solr 9.x plugin
Karl Wright created CONNECTORS-1723: --- Summary: Need Solr 9.x plugin Key: CONNECTORS-1723 URL: https://issues.apache.org/jira/browse/CONNECTORS-1723 Project: ManifoldCF Issue Type: New Feature Reporter: Karl Wright Assignee: Karl Wright Please attach patch with code we will use for plugin. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CONNECTORS-1541) Documents updated in Google Drive are send with 0 byte to CMIS Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1541. - Resolution: Fixed r1902902. In the future, it's really best to create a followup ticket if the original fixed ticket isn't sufficient. Otherwise we cannot easily track what's been done in what release. > Documents updated in Google Drive are send with 0 byte to CMIS Output > Connector > --- > > Key: CONNECTORS-1541 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1541 > Project: ManifoldCF > Issue Type: Bug > Components: CMIS Output Connector >Affects Versions: ManifoldCF 2.10 >Reporter: Douglas C. R. Paes >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.12 > > Attachments: CmisOutputConnector.java > > > When dealing with migration process, like when using the CMIS Output > Connector to ingest content into an ECM (Alfresco in my case), I noticed that > when a document is updated inside Google Drive, the engine is able to detect > the change and put it into the queue to be updated into the output. > By using the CMIS Output Connector, the document is versioned into Alfresco, > but this new version is always created as a 0 byte file. > > I configured the issue as a *Framework core* because I am still not sure who > is causing the problem, if the *GoogleDrive Connector* or the *CMIS Output > Connector* -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1541) Documents updated in Google Drive are send with 0 byte to CMIS Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17569300#comment-17569300 ] Karl Wright commented on CONNECTORS-1541: - This looks good now. Will pull it in shortly. Thanks!!! > Documents updated in Google Drive are send with 0 byte to CMIS Output > Connector > --- > > Key: CONNECTORS-1541 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1541 > Project: ManifoldCF > Issue Type: Bug > Components: CMIS Output Connector >Affects Versions: ManifoldCF 2.10 >Reporter: Douglas C. R. Paes >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.12 > > Attachments: CmisOutputConnector.java > > > When dealing with migration process, like when using the CMIS Output > Connector to ingest content into an ECM (Alfresco in my case), I noticed that > when a document is updated inside Google Drive, the engine is able to detect > the change and put it into the queue to be updated into the output. > By using the CMIS Output Connector, the document is versioned into Alfresco, > but this new version is always created as a 0 byte file. > > I configured the issue as a *Framework core* because I am still not sure who > is causing the problem, if the *GoogleDrive Connector* or the *CMIS Output > Connector* -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1722) remove xalan dependency due to it being end of life
[ https://issues.apache.org/jira/browse/CONNECTORS-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568746#comment-17568746 ] Karl Wright commented on CONNECTORS-1722: - Xalan is a downstream dependency of many libraries referenced by ManifoldCF connectors. Try mvn dependency:tree to see these. If the java runtime contains it as stated, please suggest a TESTED patch for changing the necessary build.xml downloads and references. > remove xalan dependency due to it being end of life > --- > > Key: CONNECTORS-1722 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1722 > Project: ManifoldCF > Issue Type: Improvement >Reporter: PJ Fanning >Priority: Major > > Xalan is no longer supported. > https://lists.apache.org/thread/s8kjny5270ssfcp46v0fl39lk98987w7 > It is better to use JAXP TransformerFactory than using xalan directly. If you > add xalan dependency just to ensure that you have a JAXP compliant > transformer on the classpath, this is unnecessary - the Java runtime has a > built-in implementation. > See https://github.com/apache/manifoldcf/pull/130 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1541) Documents updated in Google Drive are send with 0 byte to CMIS Output Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17567382#comment-17567382 ] Karl Wright commented on CONNECTORS-1541: - [~douglascrp], we cannot accept the patch as written because it changes the formatting of the entire files being touched from 2 spaces per indent to 4. This makes it impossible to see the actual diff. Can you resubmit without changing the formatting please? > Documents updated in Google Drive are send with 0 byte to CMIS Output > Connector > --- > > Key: CONNECTORS-1541 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1541 > Project: ManifoldCF > Issue Type: Bug > Components: CMIS Output Connector >Affects Versions: ManifoldCF 2.10 >Reporter: Douglas C. R. Paes >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.12 > > Attachments: CmisOutputConnector.java > > > When dealing with migration process, like when using the CMIS Output > Connector to ingest content into an ECM (Alfresco in my case), I noticed that > when a document is updated inside Google Drive, the engine is able to detect > the change and put it into the queue to be updated into the output. > By using the CMIS Output Connector, the document is versioned into Alfresco, > but this new version is always created as a 0 byte file. > > I configured the issue as a *Framework core* because I am still not sure who > is causing the problem, if the *GoogleDrive Connector* or the *CMIS Output > Connector* -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CONNECTORS-1720) Fatal Error due to NPE in RepriorizationTracker
[
https://issues.apache.org/jira/browse/CONNECTORS-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1720.
-
Resolution: Fixed
r1902717
> Fatal Error due to NPE in RepriorizationTracker
> ---
>
> Key: CONNECTORS-1720
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1720
> Project: ManifoldCF
> Issue Type: Bug
>Affects Versions: ManifoldCF 2.22
>Reporter: Maunel Bach
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.23
>
>
> A couple of times a week our ManifoldCF instance logged a stacktrace like
> that:
> {code:bash}
> 2022-07-06T14:04:39,297 FATAL [Set priority thread]
> org.apache.manifoldcf.crawlerthreads: Error tossed: null
> java.lang.NullPointerException: null
> at
> org.apache.manifoldcf.crawler.reprioritizationtracker.ReprioritizationTracker$PreloadKey.hashCode(ReprioritizationTracker.java:460)
> ~[mcf-pull-agent.jar:?]
> at java.util.HashMap.hash(HashMap.java:339) ~[?:?]
> at java.util.HashMap.get(HashMap.java:552) ~[?:?]
> at
> org.apache.manifoldcf.crawler.reprioritizationtracker.ReprioritizationTracker.addPreloadRequest(ReprioritizationTracker.java:234)
> ~[mcf-pull-agent.jar:?]
> at
> org.apache.manifoldcf.crawler.system.PriorityCalculator.makePreloadRequest(PriorityCalculator.java:123)
> ~[mcf-pull-agent.jar:?]
> at
> org.apache.manifoldcf.crawler.system.ManifoldCF.writeDocumentPriorities(ManifoldCF.java:1247)
> ~[mcf-pull-agent.jar:?]
> at
> org.apache.manifoldcf.crawler.system.SetPriorityThread.run(SetPriorityThread.java:141)
> ~[mcf-pull-agent.jar:?]{code}
> We tracked it down to a not null safe {{hashCode()}} implementation of the
> private {{PreloadRequest}} class consuming the first entry of the array
> {{binNames}}. The specifics of the array is an implementation detail of a
> connector base class. Potentially array might contain {{null}} values at
> least for the {{JiraRepositoryConnector}} and more critical for the
> {{WebcrawlerConnector}}.
> As a solution we fixed the {{hashCode()}} implementation in our own code base
> which did the trick. We should consider to fix this bug in the Apache
> repository as well. But of course the fix does not help improve the
> extraction and usage of the {{binName}} itself.
> We stumbled over this bug because the jobs on our ManifoldCF instance got
> stuck. Though we cannot be sure whether there was a correlation between this
> greater incident and the fatal error here because that was not the only fatal
> bug we fixed to get ManifoldCF up an running again. But the topic
> "repriorization" indicates that a bit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1720) Fatal Error due to NPE in RepriorizationTracker
[
https://issues.apache.org/jira/browse/CONNECTORS-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17566747#comment-17566747
]
Karl Wright commented on CONNECTORS-1720:
-
[~mil7], ok, now it is clearer what the failure mode is.
The binNames[] array comes from the connector. The connector is supposed to
insure that they make sense in the context of throttling. A binName that is
null or empty is not expected because it basically means the connector is doing
something strange. The WebConnector, which you referenced, uses the domain
name as a bin name, so any URL that doesn't have a domain name isn't in fact a
valid URL. I'd be very interested to see how that could happen given this.
> Fatal Error due to NPE in RepriorizationTracker
> ---
>
> Key: CONNECTORS-1720
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1720
> Project: ManifoldCF
> Issue Type: Bug
>Affects Versions: ManifoldCF 2.22
>Reporter: Maunel Bach
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.23
>
>
> A couple of times a week our ManifoldCF instance logged a stacktrace like
> that:
> {code:bash}
> 2022-07-06T14:04:39,297 FATAL [Set priority thread]
> org.apache.manifoldcf.crawlerthreads: Error tossed: null
> java.lang.NullPointerException: null
> at
> org.apache.manifoldcf.crawler.reprioritizationtracker.ReprioritizationTracker$PreloadKey.hashCode(ReprioritizationTracker.java:460)
> ~[mcf-pull-agent.jar:?]
> at java.util.HashMap.hash(HashMap.java:339) ~[?:?]
> at java.util.HashMap.get(HashMap.java:552) ~[?:?]
> at
> org.apache.manifoldcf.crawler.reprioritizationtracker.ReprioritizationTracker.addPreloadRequest(ReprioritizationTracker.java:234)
> ~[mcf-pull-agent.jar:?]
> at
> org.apache.manifoldcf.crawler.system.PriorityCalculator.makePreloadRequest(PriorityCalculator.java:123)
> ~[mcf-pull-agent.jar:?]
> at
> org.apache.manifoldcf.crawler.system.ManifoldCF.writeDocumentPriorities(ManifoldCF.java:1247)
> ~[mcf-pull-agent.jar:?]
> at
> org.apache.manifoldcf.crawler.system.SetPriorityThread.run(SetPriorityThread.java:141)
> ~[mcf-pull-agent.jar:?]{code}
> We tracked it down to a not null safe {{hashCode()}} implementation of the
> private {{PreloadRequest}} class consuming the first entry of the array
> {{binNames}}. The specifics of the array is an implementation detail of a
> connector base class. Potentially array might contain {{null}} values at
> least for the {{JiraRepositoryConnector}} and more critical for the
> {{WebcrawlerConnector}}.
> As a solution we fixed the {{hashCode()}} implementation in our own code base
> which did the trick. We should consider to fix this bug in the Apache
> repository as well. But of course the fix does not help improve the
> extraction and usage of the {{binName}} itself.
> We stumbled over this bug because the jobs on our ManifoldCF instance got
> stuck. Though we cannot be sure whether there was a correlation between this
> greater incident and the fatal error here because that was not the only fatal
> bug we fixed to get ManifoldCF up an running again. But the topic
> "repriorization" indicates that a bit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CONNECTORS-1720) Fatal Error due to NPE in RepriorizationTracker
[
https://issues.apache.org/jira/browse/CONNECTORS-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1720.
-
Fix Version/s: ManifoldCF 2.23
Resolution: Fixed
r1902684
> Fatal Error due to NPE in RepriorizationTracker
> ---
>
> Key: CONNECTORS-1720
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1720
> Project: ManifoldCF
> Issue Type: Bug
>Affects Versions: ManifoldCF 2.22
>Reporter: Maunel Bach
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.23
>
>
> A couple of times a week our ManifoldCF instance logged a stacktrace like
> that:
> {code:bash}
> 2022-07-06T14:04:39,297 FATAL [Set priority thread]
> org.apache.manifoldcf.crawlerthreads: Error tossed: null
> java.lang.NullPointerException: null
> at
> org.apache.manifoldcf.crawler.reprioritizationtracker.ReprioritizationTracker$PreloadKey.hashCode(ReprioritizationTracker.java:460)
> ~[mcf-pull-agent.jar:?]
> at java.util.HashMap.hash(HashMap.java:339) ~[?:?]
> at java.util.HashMap.get(HashMap.java:552) ~[?:?]
> at
> org.apache.manifoldcf.crawler.reprioritizationtracker.ReprioritizationTracker.addPreloadRequest(ReprioritizationTracker.java:234)
> ~[mcf-pull-agent.jar:?]
> at
> org.apache.manifoldcf.crawler.system.PriorityCalculator.makePreloadRequest(PriorityCalculator.java:123)
> ~[mcf-pull-agent.jar:?]
> at
> org.apache.manifoldcf.crawler.system.ManifoldCF.writeDocumentPriorities(ManifoldCF.java:1247)
> ~[mcf-pull-agent.jar:?]
> at
> org.apache.manifoldcf.crawler.system.SetPriorityThread.run(SetPriorityThread.java:141)
> ~[mcf-pull-agent.jar:?]{code}
> We tracked it down to a not null safe {{hashCode()}} implementation of the
> private {{PreloadRequest}} class consuming the first entry of the array
> {{binNames}}. The specifics of the array is an implementation detail of a
> connector base class. Potentially array might contain {{null}} values at
> least for the {{JiraRepositoryConnector}} and more critical for the
> {{WebcrawlerConnector}}.
> As a solution we fixed the {{hashCode()}} implementation in our own code base
> which did the trick. We should consider to fix this bug in the Apache
> repository as well. But of course the fix does not help improve the
> extraction and usage of the {{binName}} itself.
> We stumbled over this bug because the jobs on our ManifoldCF instance got
> stuck. Though we cannot be sure whether there was a correlation between this
> greater incident and the fatal error here because that was not the only fatal
> bug we fixed to get ManifoldCF up an running again. But the topic
> "repriorization" indicates that a bit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CONNECTORS-1720) Fatal Error due to NPE in RepriorizationTracker
[
https://issues.apache.org/jira/browse/CONNECTORS-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1720:
---
Assignee: Karl Wright
> Fatal Error due to NPE in RepriorizationTracker
> ---
>
> Key: CONNECTORS-1720
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1720
> Project: ManifoldCF
> Issue Type: Bug
>Affects Versions: ManifoldCF 2.22
>Reporter: Maunel Bach
>Assignee: Karl Wright
>Priority: Major
>
> A couple of times a week our ManifoldCF instance logged a stacktrace like
> that:
> {code:bash}
> 2022-07-06T14:04:39,297 FATAL [Set priority thread]
> org.apache.manifoldcf.crawlerthreads: Error tossed: null
> java.lang.NullPointerException: null
> at
> org.apache.manifoldcf.crawler.reprioritizationtracker.ReprioritizationTracker$PreloadKey.hashCode(ReprioritizationTracker.java:460)
> ~[mcf-pull-agent.jar:?]
> at java.util.HashMap.hash(HashMap.java:339) ~[?:?]
> at java.util.HashMap.get(HashMap.java:552) ~[?:?]
> at
> org.apache.manifoldcf.crawler.reprioritizationtracker.ReprioritizationTracker.addPreloadRequest(ReprioritizationTracker.java:234)
> ~[mcf-pull-agent.jar:?]
> at
> org.apache.manifoldcf.crawler.system.PriorityCalculator.makePreloadRequest(PriorityCalculator.java:123)
> ~[mcf-pull-agent.jar:?]
> at
> org.apache.manifoldcf.crawler.system.ManifoldCF.writeDocumentPriorities(ManifoldCF.java:1247)
> ~[mcf-pull-agent.jar:?]
> at
> org.apache.manifoldcf.crawler.system.SetPriorityThread.run(SetPriorityThread.java:141)
> ~[mcf-pull-agent.jar:?]{code}
> We tracked it down to a not null safe {{hashCode()}} implementation of the
> private {{PreloadRequest}} class consuming the first entry of the array
> {{binNames}}. The specifics of the array is an implementation detail of a
> connector base class. Potentially array might contain {{null}} values at
> least for the {{JiraRepositoryConnector}} and more critical for the
> {{WebcrawlerConnector}}.
> As a solution we fixed the {{hashCode()}} implementation in our own code base
> which did the trick. We should consider to fix this bug in the Apache
> repository as well. But of course the fix does not help improve the
> extraction and usage of the {{binName}} itself.
> We stumbled over this bug because the jobs on our ManifoldCF instance got
> stuck. Though we cannot be sure whether there was a correlation between this
> greater incident and the fatal error here because that was not the only fatal
> bug we fixed to get ManifoldCF up an running again. But the topic
> "repriorization" indicates that a bit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1713) JIRA Repository Connector ignores issue security when ingesting from JIRA 8.20+
[
https://issues.apache.org/jira/browse/CONNECTORS-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17553253#comment-17553253
]
Karl Wright commented on CONNECTORS-1713:
-
[~schuch], the SharePoint connector requires configuration information that
specifies the version you're trying to talk to. But that's because there isn't
a version interrogation present in the API for SharePoint. If there is such a
capability for Jira I'd see how difficult it would be to use it. The version
information should ideally be retrieved during the connect() method, and thus
stored as an instance variable for the connector. Make sure you clean it out
again when disconnect() is called though to prevent a pooled instance from
malfunctioning.
> JIRA Repository Connector ignores issue security when ingesting from JIRA
> 8.20+
> ---
>
> Key: CONNECTORS-1713
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1713
> Project: ManifoldCF
> Issue Type: Bug
> Components: JIRA connector
>Affects Versions: ManifoldCF 2.22
>Reporter: Markus Schuch
>Priority: Major
> Attachments: api-docs.png
>
>
> There was obviously a change in the behaviour of the JIRA Server REST API:
> The {{GET /rest/user/viewissue/search}} has a parameter {{username}}.
> In JIRA 8.13.x the value must be to double quotes ({{username=""}}) to fetch
> all users that have browse permission for the issue.
> In JIRA 8.20.x the value must be empty ({{username=}}).
> I found no information about this change in the JIRA Release Notes.
> I raised a question in the Atlassian Dev Community:
> https://community.developer.atlassian.com/t/rest-api-change-in-behaviour-of-find-users-with-browse-permission-get-rest-user-viewissue-search/58819
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (CONNECTORS-1713) JIRA Repository Connector ignores issue security when ingesting from JIRA 8.20+
[
https://issues.apache.org/jira/browse/CONNECTORS-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552637#comment-17552637
]
Karl Wright commented on CONNECTORS-1713:
-
[~schuch], are you saying the versions prior to 8.20 would be unsupported? Or
is it just that we don't know?
If we don't know, I'd say go ahead and make the needed repairs. If we DO know
that it would break, we could make the (new) behavior contingent on the results
of a version query to JIRA, correct?
> JIRA Repository Connector ignores issue security when ingesting from JIRA
> 8.20+
> ---
>
> Key: CONNECTORS-1713
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1713
> Project: ManifoldCF
> Issue Type: Bug
> Components: JIRA connector
>Affects Versions: ManifoldCF 2.22
>Reporter: Markus Schuch
>Priority: Major
> Attachments: api-docs.png
>
>
> There was obviously a change in the behaviour of the JIRA Server REST API:
> The {{GET /rest/user/viewissue/search}} has a parameter {{username}}.
> In JIRA 8.13.x the value must be to double quotes ({{username=""}}) to fetch
> all users that have browse permission for the issue.
> In JIRA 8.20.x the value must be empty ({{username=}}).
> I found no information about this change in the JIRA Release Notes.
> I raised a question in the Atlassian Dev Community:
> https://community.developer.atlassian.com/t/rest-api-change-in-behaviour-of-find-users-with-browse-permission-get-rest-user-viewissue-search/58819
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (CONNECTORS-1717) upgrade to connectors 2.17.2
[ https://issues.apache.org/jira/browse/CONNECTORS-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1717: --- Assignee: Karl Wright > upgrade to connectors 2.17.2 > > > Key: CONNECTORS-1717 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1717 > Project: ManifoldCF > Issue Type: Improvement >Affects Versions: ManifoldCF 2.22 >Reporter: PJ Fanning >Assignee: Karl Wright >Priority: Major > Attachments: patch.txt > > > ant build.xml has version 2.15.0 > pom.xml has 2.17.0 > chemistry dependency seems to have log4j 2.17.2 > there are issues in versions up to v2.17.1 > seems best to standardise on 2.17.2 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Comment Edited] (CONNECTORS-1715) Vulnerabilities in 45 jars in Apache Manifold CF 2.22.1 version
[ https://issues.apache.org/jira/browse/CONNECTORS-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552149#comment-17552149 ] Karl Wright edited comment on CONNECTORS-1715 at 6/9/22 11:50 AM: -- [~pj.fanning], this is a blanket scan identifying jars with known CVEs. There has been no analysis done whatsoever about whether the specific CVE attack is even a possibility in the ManifoldCF environment. That's a lot of work but I will wager after all of that the major problem is that the tool doesn't understand the actual usage of ManifoldCF and is thus incapable of giving good advice. Another thing to note is that most of ManifoldCF's dependencies come from Tika. We just upgraded a month ago to the latest Tika 1.x version, which required massive dependency updates precisely to address CVEs that had been noted. This took me almost three weeks because many of the underlying contracts in the jars also had to be updated. That's a lot of work if a vulnerability cannot in fact be exploited at all, just to make a dumb tool happy. I think it's fine if a careful analysis is done and an ACTUAL vulnerability is detected, but we want to not be stupid about this. Can't afford it. was (Author: kwri...@metacarta.com): [~pj.fanning], this is a blanket scan identifying jars with known CVEs. There has been no analysis done whatsoever about whether the specific CVE attack is even a possibility in the ManifoldCF environment. That's a lot of work but I will wager after all of that the major problem is that the tool doesn't understand the actual usage of ManifoldCF and is thus incapable of giving good advice. > Vulnerabilities in 45 jars in Apache Manifold CF 2.22.1 version > --- > > Key: CONNECTORS-1715 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1715 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.22 >Reporter: Himanshu >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.23 > > Attachments: dependency-check-report-Apache Manifold.html > > > 45 vulnerable jars are present in apache-manifoldcf version 2.22.1 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CONNECTORS-1715) Vulnerabilities in 45 jars in Apache Manifold CF 2.22.1 version
[ https://issues.apache.org/jira/browse/CONNECTORS-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552149#comment-17552149 ] Karl Wright commented on CONNECTORS-1715: - [~pj.fanning], this is a blanket scan identifying jars with known CVEs. There has been no analysis done whatsoever about whether the specific CVE attack is even a possibility in the ManifoldCF environment. That's a lot of work but I will wager after all of that the major problem is that the tool doesn't understand the actual usage of ManifoldCF and is thus incapable of giving good advice. > Vulnerabilities in 45 jars in Apache Manifold CF 2.22.1 version > --- > > Key: CONNECTORS-1715 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1715 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.22 >Reporter: Himanshu >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.23 > > Attachments: dependency-check-report-Apache Manifold.html > > > 45 vulnerable jars are present in apache-manifoldcf version 2.22.1 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CONNECTORS-1714) upgrade commons-beanutils due to CVE
[ https://issues.apache.org/jira/browse/CONNECTORS-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552132#comment-17552132 ] Karl Wright commented on CONNECTORS-1714: - r1901777 > upgrade commons-beanutils due to CVE > > > Key: CONNECTORS-1714 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1714 > Project: ManifoldCF > Issue Type: Bug >Reporter: PJ Fanning >Assignee: Karl Wright >Priority: Major > > https://github.com/advisories/GHSA-6phf-73q6-gh87 > https://github.com/apache/manifoldcf/blob/trunk/connectors/alfresco/pom.xml > -- upgrade beanutils to v1.9.4 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (CONNECTORS-1714) upgrade commons-beanutils due to CVE
[ https://issues.apache.org/jira/browse/CONNECTORS-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1714. - Fix Version/s: ManifoldCF 2.23 Resolution: Fixed Thanks, [~pj.fanning]! > upgrade commons-beanutils due to CVE > > > Key: CONNECTORS-1714 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1714 > Project: ManifoldCF > Issue Type: Bug >Reporter: PJ Fanning >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.23 > > > https://github.com/advisories/GHSA-6phf-73q6-gh87 > https://github.com/apache/manifoldcf/blob/trunk/connectors/alfresco/pom.xml > -- upgrade beanutils to v1.9.4 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CONNECTORS-1716) should not use http to download artifacts (need https)
[
https://issues.apache.org/jira/browse/CONNECTORS-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552090#comment-17552090
]
Karl Wright commented on CONNECTORS-1716:
-
Tried that URL here and it worked fine via "ant download-nuxeo-client":
{code}
BUILD SUCCESSFUL
Total time: 5 seconds
{code}
So I committed that change.
> should not use http to download artifacts (need https)
> --
>
> Key: CONNECTORS-1716
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1716
> Project: ManifoldCF
> Issue Type: Bug
>Reporter: PJ Fanning
>Assignee: Karl Wright
>Priority: Major
>
> build.xml has a number of insecure http URLs
> the nexus one is a special problem because the https equivalent has the wrong
> SSL cert - see https://maven.nuxeo.com/
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (CONNECTORS-1716) should not use http to download artifacts (need https)
[ https://issues.apache.org/jira/browse/CONNECTORS-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552086#comment-17552086 ] Karl Wright commented on CONNECTORS-1716: - That sounds like a workable option for the moment. I'll commit it if it seems to download the artifacts needed correctly. > should not use http to download artifacts (need https) > -- > > Key: CONNECTORS-1716 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1716 > Project: ManifoldCF > Issue Type: Bug >Reporter: PJ Fanning >Assignee: Karl Wright >Priority: Major > > build.xml has a number of insecure http URLs > the nexus one is a special problem because the https equivalent has the wrong > SSL cert - see https://maven.nuxeo.com/ -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CONNECTORS-1714) upgrade commons-beanutils due to CVE
[ https://issues.apache.org/jira/browse/CONNECTORS-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552084#comment-17552084 ] Karl Wright commented on CONNECTORS-1714: - [~pj.fanning], I did not configure or set up the Travis build and it seems like it was broken before you started. But I did look at the patch and it doesn't look to me like it's complete. I have a busy workday today so I won't be able to spend hours on this but I do need to point out that the Maven build for MCF is secondary, not primary, so you first want to make sure that the ant build is correct in any case. Thanks! > upgrade commons-beanutils due to CVE > > > Key: CONNECTORS-1714 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1714 > Project: ManifoldCF > Issue Type: Bug >Reporter: PJ Fanning >Assignee: Karl Wright >Priority: Major > > https://github.com/advisories/GHSA-6phf-73q6-gh87 > https://github.com/apache/manifoldcf/blob/trunk/connectors/alfresco/pom.xml > -- upgrade beanutils to v1.9.4 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CONNECTORS-1716) should not use http to download artifacts (need https)
[ https://issues.apache.org/jira/browse/CONNECTORS-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552079#comment-17552079 ] Karl Wright commented on CONNECTORS-1716: - I looked into the nuxeo case; this is a client library required for building the nuxeo connector, so it is not just a testing artifact (where I have encountered poor adherence to ssl use in the past). Unfortunately, as you point out, nuxeo's cert has expired and the only solution is to fully decommission the Nuxeo connector until this has been addressed. > should not use http to download artifacts (need https) > -- > > Key: CONNECTORS-1716 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1716 > Project: ManifoldCF > Issue Type: Bug >Reporter: PJ Fanning >Assignee: Karl Wright >Priority: Major > > build.xml has a number of insecure http URLs > the nexus one is a special problem because the https equivalent has the wrong > SSL cert - see https://maven.nuxeo.com/ -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CONNECTORS-1716) should not use http to download artifacts (need https)
[ https://issues.apache.org/jira/browse/CONNECTORS-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552074#comment-17552074 ] Karl Wright commented on CONNECTORS-1716: - [~pj.fanning] I am well aware of the reason. The only option, however, is to turn off the specific integration test. Please note that the integration tests are NOT run until after the build takes place, so they will not have any potential of corrupting the build. Testing artifacts are not distributed as part of the distribution either. > should not use http to download artifacts (need https) > -- > > Key: CONNECTORS-1716 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1716 > Project: ManifoldCF > Issue Type: Bug >Reporter: PJ Fanning >Assignee: Karl Wright >Priority: Major > > build.xml has a number of insecure http URLs > the nexus one is a special problem because the https equivalent has the wrong > SSL cert - see https://maven.nuxeo.com/ -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Comment Edited] (CONNECTORS-1716) should not use http to download artifacts (need https)
[ https://issues.apache.org/jira/browse/CONNECTORS-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552065#comment-17552065 ] Karl Wright edited comment on CONNECTORS-1716 at 6/9/22 9:15 AM: - Please propose a patch. I am not certain what downloads you are specifically talking about but this happens only during the build process, and may be for a testing artifact. In some cases https is not used because it is not supported by the test component supplier, e.g. alfresco or mongodb. If you can find a way around that, great, if not there is nothing we can do about it - e.g. the nuxeo case. You should file a ticket with them. was (Author: kwri...@metacarta.com): Please propose a patch. I am not certain what downloads you are specifically talking about but this happens only during the build process, and may be for a testing artifact. In some cases https is not used because it is not supported by the test component supplier, e.g. alfresco or mongodb. If you can find a way around that, great, if not there is nothing we can do about it. > should not use http to download artifacts (need https) > -- > > Key: CONNECTORS-1716 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1716 > Project: ManifoldCF > Issue Type: Bug >Reporter: PJ Fanning >Assignee: Karl Wright >Priority: Major > > build.xml has a number of insecure http URLs > the nexus one is a special problem because the https equivalent has the wrong > SSL cert - see https://maven.nuxeo.com/ -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CONNECTORS-1716) should not use http to download artifacts (need https)
[ https://issues.apache.org/jira/browse/CONNECTORS-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552065#comment-17552065 ] Karl Wright commented on CONNECTORS-1716: - Please propose a patch. I am not certain what downloads you are specifically talking about but this happens only during the build process, and may be for a testing artifact. In some cases https is not used because it is not supported by the test component supplier, e.g. alfresco or mongodb. If you can find a way around that, great, if not there is nothing we can do about it. > should not use http to download artifacts (need https) > -- > > Key: CONNECTORS-1716 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1716 > Project: ManifoldCF > Issue Type: Bug >Reporter: PJ Fanning >Assignee: Karl Wright >Priority: Major > > build.xml has a number of insecure http URLs > the nexus one is a special problem because the https equivalent has the wrong > SSL cert - see https://maven.nuxeo.com/ -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (CONNECTORS-1716) should not use http to download artifacts (need https)
[ https://issues.apache.org/jira/browse/CONNECTORS-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1716: --- Assignee: Karl Wright > should not use http to download artifacts (need https) > -- > > Key: CONNECTORS-1716 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1716 > Project: ManifoldCF > Issue Type: Bug >Reporter: PJ Fanning >Assignee: Karl Wright >Priority: Major > > build.xml has a number of insecure http URLs > the nexus one is a special problem because the https equivalent has the wrong > SSL cert - see https://maven.nuxeo.com/ -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CONNECTORS-1714) upgrade commons-beanutils due to CVE
[ https://issues.apache.org/jira/browse/CONNECTORS-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552060#comment-17552060 ] Karl Wright commented on CONNECTORS-1714: - Please propose a patch. Thanks. > upgrade commons-beanutils due to CVE > > > Key: CONNECTORS-1714 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1714 > Project: ManifoldCF > Issue Type: Bug >Reporter: PJ Fanning >Assignee: Karl Wright >Priority: Major > > https://github.com/advisories/GHSA-6phf-73q6-gh87 > https://github.com/apache/manifoldcf/blob/trunk/connectors/alfresco/pom.xml > -- upgrade beanutils to v1.9.4 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (CONNECTORS-1714) upgrade commons-beanutils due to CVE
[ https://issues.apache.org/jira/browse/CONNECTORS-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1714: --- Assignee: Karl Wright > upgrade commons-beanutils due to CVE > > > Key: CONNECTORS-1714 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1714 > Project: ManifoldCF > Issue Type: Bug >Reporter: PJ Fanning >Assignee: Karl Wright >Priority: Major > > https://github.com/advisories/GHSA-6phf-73q6-gh87 > https://github.com/apache/manifoldcf/blob/trunk/connectors/alfresco/pom.xml > -- upgrade beanutils to v1.9.4 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (CONNECTORS-1715) Vulnerabilities in 45 jars in Apache Manifold CF 2.22.1 version
[ https://issues.apache.org/jira/browse/CONNECTORS-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1715: --- Assignee: Karl Wright > Vulnerabilities in 45 jars in Apache Manifold CF 2.22.1 version > --- > > Key: CONNECTORS-1715 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1715 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.22 >Reporter: Himanshu >Assignee: Karl Wright >Priority: Major > Attachments: dependency-check-report-Apache Manifold.html > > > 45 vulnerable jars are present in apache-manifoldcf version 2.22.1 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (CONNECTORS-1715) Vulnerabilities in 45 jars in Apache Manifold CF 2.22.1 version
[ https://issues.apache.org/jira/browse/CONNECTORS-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1715. - Fix Version/s: ManifoldCF 2.23 Resolution: Won't Fix > Vulnerabilities in 45 jars in Apache Manifold CF 2.22.1 version > --- > > Key: CONNECTORS-1715 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1715 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.22 >Reporter: Himanshu >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.23 > > Attachments: dependency-check-report-Apache Manifold.html > > > 45 vulnerable jars are present in apache-manifoldcf version 2.22.1 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CONNECTORS-1715) Vulnerabilities in 45 jars in Apache Manifold CF 2.22.1 version
[ https://issues.apache.org/jira/browse/CONNECTORS-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552059#comment-17552059 ] Karl Wright commented on CONNECTORS-1715: - Sorry, most of these cannot be upgraded because there is nothing to upgrade to. Example: Axis jars. A quick look shows that the kinds of attacks listed here are operating modes for the jars in question that would make the attack vector impossible to exploit in ManifoldCF. ManifoldCF indexes data from/to trusted systems, so an attack on ManifoldCF itself from such a setup would have to involve a man-in-the-middle, which can trivially be avoided if you are on either a secure network or use Https for your connections to your repositories. ManifoldCF's UI and API we recommend also be localized to an internal network, but in any case they are what we secure. Database connection security is left as an exercise for the user; it's beyond the scope of the ManifoldCF project. > Vulnerabilities in 45 jars in Apache Manifold CF 2.22.1 version > --- > > Key: CONNECTORS-1715 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1715 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.22 >Reporter: Himanshu >Priority: Major > Attachments: dependency-check-report-Apache Manifold.html > > > 45 vulnerable jars are present in apache-manifoldcf version 2.22.1 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CONNECTORS-1667) New Tika Service Connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17547939#comment-17547939 ] Karl Wright commented on CONNECTORS-1667: - [~cguzel], this ticket is about an EXTERNAL service where Tika runs as a separate stand-alone process, and the connector communicates to it. I don't think there is any difference from a service standpoint whether you run Tika 1.x or 2.x as that service - the protocol is likely the same, although I haven't researched it. What you seem to be thinking is that the internal Tika connector should go to Tika 2.0. This is a major, major deal because most of the connector dependencies we have to update are due to Tika. I looked at it and found we'd need 4-5 weeks of a dedicated individual to do the port. Are you volunteering? If so I can advise you. Otherwise we will be staying current with Tika 1.x releases for now, and that is all. > New Tika Service Connector > -- > > Key: CONNECTORS-1667 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1667 > Project: ManifoldCF > Issue Type: New Feature > Components: Tika service connector >Reporter: Julien Massiera >Assignee: Julien Massiera >Priority: Major > Fix For: ManifoldCF 2.20 > > > The current Tika Service Connector exploits the '/unpack/all' endpoint of a > Tika Server. This endpoint is not optimal to only extract document's metadata > and content. We should develop a new connector based on the 'rmeta' endpoint > which is more suited for our needs. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (CONNECTORS-1712) Broken Velocity UI
[
https://issues.apache.org/jira/browse/CONNECTORS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1712.
-
Fix Version/s: ManifoldCF 2.23
ManifoldCF 2.22.1
Resolution: Fixed
> Broken Velocity UI
> --
>
> Key: CONNECTORS-1712
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1712
> Project: ManifoldCF
> Issue Type: Bug
> Components: API
>Affects Versions: ManifoldCF 2.22
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
> Fix For: ManifoldCF 2.23, ManifoldCF 2.22.1
>
>
> In the mcf-crawler-ui, we cannot enter in edition mode for any connector
> because there is a problem with Velocity.
> We obtain the following error in the logs:
>
> {code:java}
> java.lang.NoSuchMethodError: 'void
> org.apache.velocity.app.VelocityEngine.setExtendedProperties(org.apache.commons.collections.ExtendedProperties)'
> at
> org.apache.manifoldcf.core.i18n.Messages.createVelocityEngine(Messages.java:62)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.outputResourceWithVelocity(Messages.java:132)
> ~[mcf-ui-core.jar:?]
> at
> com.francelabs.datafari.connectors.share.Messages.outputResourceWithVelocity(Messages.java:111)
> ~[?:?]
> at
> com.francelabs.datafari.connectors.share.SharedDriveConnector.outputSpecificationHeader(SharedDriveConnector.java:2829)
> ~[?:?]
> at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:977)
> ~[mcf-crawler-ui.jar:?]
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> ~[jasper.jar:9.0.56]
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> ~[jasper.jar:9.0.56]
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> ~[tomcat-websocket.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
> [catalina.jar:9.0.56]
> at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
> [tomcat-util.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
> [tomcat-util.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> [tomcat-util.jar:9.0.56]
> at java.lang.Thread.run(Thread.java:829) [?:?] {code}
>
>
> After some investigations it
[jira] [Commented] (CONNECTORS-1712) Broken Velocity UI
[
https://issues.apache.org/jira/browse/CONNECTORS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533542#comment-17533542
]
Karl Wright commented on CONNECTORS-1712:
-
Committed r1900711 which I confirmed does fix the problem.
The functionality that was there before still exists but you cannot do it with
a simple "setConfiguration". You have to add a property afterwards with
another command. I could only find this by inspecting the code.
> Broken Velocity UI
> --
>
> Key: CONNECTORS-1712
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1712
> Project: ManifoldCF
> Issue Type: Bug
> Components: API
>Affects Versions: ManifoldCF 2.22
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
>
> In the mcf-crawler-ui, we cannot enter in edition mode for any connector
> because there is a problem with Velocity.
> We obtain the following error in the logs:
>
> {code:java}
> java.lang.NoSuchMethodError: 'void
> org.apache.velocity.app.VelocityEngine.setExtendedProperties(org.apache.commons.collections.ExtendedProperties)'
> at
> org.apache.manifoldcf.core.i18n.Messages.createVelocityEngine(Messages.java:62)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.outputResourceWithVelocity(Messages.java:132)
> ~[mcf-ui-core.jar:?]
> at
> com.francelabs.datafari.connectors.share.Messages.outputResourceWithVelocity(Messages.java:111)
> ~[?:?]
> at
> com.francelabs.datafari.connectors.share.SharedDriveConnector.outputSpecificationHeader(SharedDriveConnector.java:2829)
> ~[?:?]
> at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:977)
> ~[mcf-crawler-ui.jar:?]
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> ~[jasper.jar:9.0.56]
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> ~[jasper.jar:9.0.56]
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> ~[tomcat-websocket.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
> [catalina.jar:9.0.56]
> at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
> [tomcat-util.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
> [tomcat-util.jar:9.0.56]
> at
>
[jira] [Commented] (CONNECTORS-1712) Broken Velocity UI
[
https://issues.apache.org/jira/browse/CONNECTORS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533538#comment-17533538
]
Karl Wright commented on CONNECTORS-1712:
-
[~schuch], yes, this approach will not work.
There is no other way than to get an object through that can be seen when the
Resource Loader's init(ExtProperties) method is called. There does not seem to
be documentation on how to do it if indeed it is possible. I would suggest
reviewing the Velocity 2.x codebase to see if there might be a way. If not I
propose attempting to extend the VelocityEngine class to allow the addition of
a method that does what we need.
> Broken Velocity UI
> --
>
> Key: CONNECTORS-1712
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1712
> Project: ManifoldCF
> Issue Type: Bug
> Components: API
>Affects Versions: ManifoldCF 2.22
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
>
> In the mcf-crawler-ui, we cannot enter in edition mode for any connector
> because there is a problem with Velocity.
> We obtain the following error in the logs:
>
> {code:java}
> java.lang.NoSuchMethodError: 'void
> org.apache.velocity.app.VelocityEngine.setExtendedProperties(org.apache.commons.collections.ExtendedProperties)'
> at
> org.apache.manifoldcf.core.i18n.Messages.createVelocityEngine(Messages.java:62)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.outputResourceWithVelocity(Messages.java:132)
> ~[mcf-ui-core.jar:?]
> at
> com.francelabs.datafari.connectors.share.Messages.outputResourceWithVelocity(Messages.java:111)
> ~[?:?]
> at
> com.francelabs.datafari.connectors.share.SharedDriveConnector.outputSpecificationHeader(SharedDriveConnector.java:2829)
> ~[?:?]
> at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:977)
> ~[mcf-crawler-ui.jar:?]
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> ~[jasper.jar:9.0.56]
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> ~[jasper.jar:9.0.56]
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> ~[tomcat-websocket.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
> [catalina.jar:9.0.56]
> at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
>
[jira] [Commented] (CONNECTORS-1712) Broken Velocity UI
[
https://issues.apache.org/jira/browse/CONNECTORS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533414#comment-17533414
]
Karl Wright commented on CONNECTORS-1712:
-
I committed a tentative fix to trunk: r1900677
The way this fix works is to convert the class instance to a fully-qualified
class name and pass that. Then, inside the custom resource loader, I turn the
class name back to a Class object using Class.forName(). That, of course, uses
the class loader in place at that moment, which (crosses fingers) may always be
one for which the Class can be loaded, even it it's inside a connector and we
need the connector class path loader. The VelocityContext is initialized
inside a Connector method, after all, so that should be the case.
[~julienFL], you should be able to try this for your example using the
following steps:
(1) Check out a fresh copy of mcf trunk.
(2) ant make-core-deps (should not download 1.7 anymore)
(3) ant build
(4) cd dist/example
(5) start, and try this out.
Please let me know ASAP if this is working as expected, or if there are
problems.
> Broken Velocity UI
> --
>
> Key: CONNECTORS-1712
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1712
> Project: ManifoldCF
> Issue Type: Bug
> Components: API
>Affects Versions: ManifoldCF 2.22
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
>
> In the mcf-crawler-ui, we cannot enter in edition mode for any connector
> because there is a problem with Velocity.
> We obtain the following error in the logs:
>
> {code:java}
> java.lang.NoSuchMethodError: 'void
> org.apache.velocity.app.VelocityEngine.setExtendedProperties(org.apache.commons.collections.ExtendedProperties)'
> at
> org.apache.manifoldcf.core.i18n.Messages.createVelocityEngine(Messages.java:62)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.outputResourceWithVelocity(Messages.java:132)
> ~[mcf-ui-core.jar:?]
> at
> com.francelabs.datafari.connectors.share.Messages.outputResourceWithVelocity(Messages.java:111)
> ~[?:?]
> at
> com.francelabs.datafari.connectors.share.SharedDriveConnector.outputSpecificationHeader(SharedDriveConnector.java:2829)
> ~[?:?]
> at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:977)
> ~[mcf-crawler-ui.jar:?]
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> ~[jasper.jar:9.0.56]
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> ~[jasper.jar:9.0.56]
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> ~[tomcat-websocket.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
> [catalina.jar:9.0.56]
> at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
> [tomcat-coyote.jar:9.0.56]
> at
>
[jira] [Commented] (CONNECTORS-1712) Broken Velocity UI
[
https://issues.apache.org/jira/browse/CONNECTORS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533366#comment-17533366
]
Karl Wright commented on CONNECTORS-1712:
-
This was the Velocity 1.x code:
{code}
/**
* Resource loader can be loaded either via class name or be passed
* in as an instance.
*/
ExtendedProperties configuration = (ExtendedProperties) it.next();
String loaderClass =
StringUtils.nullTrim(configuration.getString("class"));
ResourceLoader loaderInstance = (ResourceLoader)
configuration.get("instance");
if (loaderInstance != null)
{
resourceLoader = loaderInstance;
}
else if (loaderClass != null)
{
resourceLoader = ResourceLoaderFactory.getLoader(rsvc,
loaderClass);
}
else
{
String msg = "Unable to find '" +
configuration.getString(RESOURCE_LOADER_IDENTIFIER) +
".resource.loader.class' specification in
configuration." +
" This is a critical value. Please adjust
configuration.";
log.error(msg);
throw new VelocityException(msg);
}
resourceLoader.commonInit(rsvc, configuration);
resourceLoader.init(configuration);
resourceLoaders.add(resourceLoader);
{code}
Note how you had your choice of providing the class (as a string), or the
instance (as an object of type ResourceLoader). AFAICT, they simply removed
the "instance" option entirely in Velocity 2.x. We need to find a way of doing
the same thing.
> Broken Velocity UI
> --
>
> Key: CONNECTORS-1712
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1712
> Project: ManifoldCF
> Issue Type: Bug
> Components: API
>Affects Versions: ManifoldCF 2.22
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
>
> In the mcf-crawler-ui, we cannot enter in edition mode for any connector
> because there is a problem with Velocity.
> We obtain the following error in the logs:
>
> {code:java}
> java.lang.NoSuchMethodError: 'void
> org.apache.velocity.app.VelocityEngine.setExtendedProperties(org.apache.commons.collections.ExtendedProperties)'
> at
> org.apache.manifoldcf.core.i18n.Messages.createVelocityEngine(Messages.java:62)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.outputResourceWithVelocity(Messages.java:132)
> ~[mcf-ui-core.jar:?]
> at
> com.francelabs.datafari.connectors.share.Messages.outputResourceWithVelocity(Messages.java:111)
> ~[?:?]
> at
> com.francelabs.datafari.connectors.share.SharedDriveConnector.outputSpecificationHeader(SharedDriveConnector.java:2829)
> ~[?:?]
> at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:977)
> ~[mcf-crawler-ui.jar:?]
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> ~[jasper.jar:9.0.56]
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> ~[jasper.jar:9.0.56]
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> ~[tomcat-websocket.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
> [catalina.jar:9.0.56]
> at
>
[jira] [Commented] (CONNECTORS-1712) Broken Velocity UI
[
https://issues.apache.org/jira/browse/CONNECTORS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533364#comment-17533364
]
Karl Wright commented on CONNECTORS-1712:
-
See:
framework\core\src\main\java\org\apache\manifoldcf\core\i18n\Messages.java
This is where the velocity engine initialization takes place, and a Class
instance needs to make it somehow into the resource loader for this to all work
right. Any ideas? I've got a modernized version of the resource loader
updated that looks for this Class in the ExtProperties object passed into the
init() method but I don't know how I can push it into there during Velocity
engine setup. Help???
> Broken Velocity UI
> --
>
> Key: CONNECTORS-1712
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1712
> Project: ManifoldCF
> Issue Type: Bug
> Components: API
>Affects Versions: ManifoldCF 2.22
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
>
> In the mcf-crawler-ui, we cannot enter in edition mode for any connector
> because there is a problem with Velocity.
> We obtain the following error in the logs:
>
> {code:java}
> java.lang.NoSuchMethodError: 'void
> org.apache.velocity.app.VelocityEngine.setExtendedProperties(org.apache.commons.collections.ExtendedProperties)'
> at
> org.apache.manifoldcf.core.i18n.Messages.createVelocityEngine(Messages.java:62)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.outputResourceWithVelocity(Messages.java:132)
> ~[mcf-ui-core.jar:?]
> at
> com.francelabs.datafari.connectors.share.Messages.outputResourceWithVelocity(Messages.java:111)
> ~[?:?]
> at
> com.francelabs.datafari.connectors.share.SharedDriveConnector.outputSpecificationHeader(SharedDriveConnector.java:2829)
> ~[?:?]
> at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:977)
> ~[mcf-crawler-ui.jar:?]
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> ~[jasper.jar:9.0.56]
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> ~[jasper.jar:9.0.56]
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> ~[tomcat-websocket.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
> [catalina.jar:9.0.56]
> at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
>
[jira] [Commented] (CONNECTORS-1712) Broken Velocity UI
[
https://issues.apache.org/jira/browse/CONNECTORS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533362#comment-17533362
]
Karl Wright commented on CONNECTORS-1712:
-
The problem is that we allow velocity templates to be associated with a class
instance, and in order for that to work right somehow the class instance has to
get into our custom velocity resource loader. We were passing it as a
constructor argument but we no longer are allowed to instantiate it ourselves.
The velocity resource loader does receive an ExtProperties object which can
include a Class property, but I don't know how to set this in the velocity
properties so that it makes it through somehow to the resource loader.
There's also a fundamental change to how logging is done by Velocity that will
require work, but it is not insurmountable. The first problem is currently
insurmountable however and we may need to request Velocity changes to allow
this to be done again.
> Broken Velocity UI
> --
>
> Key: CONNECTORS-1712
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1712
> Project: ManifoldCF
> Issue Type: Bug
> Components: API
>Affects Versions: ManifoldCF 2.22
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
>
> In the mcf-crawler-ui, we cannot enter in edition mode for any connector
> because there is a problem with Velocity.
> We obtain the following error in the logs:
>
> {code:java}
> java.lang.NoSuchMethodError: 'void
> org.apache.velocity.app.VelocityEngine.setExtendedProperties(org.apache.commons.collections.ExtendedProperties)'
> at
> org.apache.manifoldcf.core.i18n.Messages.createVelocityEngine(Messages.java:62)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.outputResourceWithVelocity(Messages.java:132)
> ~[mcf-ui-core.jar:?]
> at
> com.francelabs.datafari.connectors.share.Messages.outputResourceWithVelocity(Messages.java:111)
> ~[?:?]
> at
> com.francelabs.datafari.connectors.share.SharedDriveConnector.outputSpecificationHeader(SharedDriveConnector.java:2829)
> ~[?:?]
> at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:977)
> ~[mcf-crawler-ui.jar:?]
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> ~[jasper.jar:9.0.56]
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> ~[jasper.jar:9.0.56]
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> ~[tomcat-websocket.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
> [catalina.jar:9.0.56]
> at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
> [tomcat-coyote.jar:9.0.56]
> at
>
[jira] [Assigned] (CONNECTORS-1712) Broken Velocity UI
[
https://issues.apache.org/jira/browse/CONNECTORS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1712:
---
Assignee: Karl Wright
> Broken Velocity UI
> --
>
> Key: CONNECTORS-1712
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1712
> Project: ManifoldCF
> Issue Type: Bug
> Components: API
>Affects Versions: ManifoldCF 2.22
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
>
> In the mcf-crawler-ui, we cannot enter in edition mode for any connector
> because there is a problem with Velocity.
> We obtain the following error in the logs:
>
> {code:java}
> java.lang.NoSuchMethodError: 'void
> org.apache.velocity.app.VelocityEngine.setExtendedProperties(org.apache.commons.collections.ExtendedProperties)'
> at
> org.apache.manifoldcf.core.i18n.Messages.createVelocityEngine(Messages.java:62)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.outputResourceWithVelocity(Messages.java:132)
> ~[mcf-ui-core.jar:?]
> at
> com.francelabs.datafari.connectors.share.Messages.outputResourceWithVelocity(Messages.java:111)
> ~[?:?]
> at
> com.francelabs.datafari.connectors.share.SharedDriveConnector.outputSpecificationHeader(SharedDriveConnector.java:2829)
> ~[?:?]
> at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:977)
> ~[mcf-crawler-ui.jar:?]
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> ~[jasper.jar:9.0.56]
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> ~[jasper.jar:9.0.56]
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> ~[tomcat-websocket.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
> [catalina.jar:9.0.56]
> at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
> [tomcat-util.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
> [tomcat-util.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> [tomcat-util.jar:9.0.56]
> at java.lang.Thread.run(Thread.java:829) [?:?] {code}
>
>
> After some investigations it seems related to the updated velocity lib from
> velocity-1.7 to velocity-engine-core-2.3
> The first problem found is that the
[jira] [Commented] (CONNECTORS-1712) Broken Velocity UI
[
https://issues.apache.org/jira/browse/CONNECTORS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533358#comment-17533358
]
Karl Wright commented on CONNECTORS-1712:
-
So what changed was that Tika now requires Velocity 2.x and our UI requires
Velocity 1.7 still.
The upgrade is nowhere near as simple as just what is stated; I'm working on it
but seems like we need to fundamentally change much more about how we set up
Velocity under 2.x.
> Broken Velocity UI
> --
>
> Key: CONNECTORS-1712
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1712
> Project: ManifoldCF
> Issue Type: Bug
> Components: API
>Affects Versions: ManifoldCF 2.22
>Reporter: Julien Massiera
>Priority: Critical
>
> In the mcf-crawler-ui, we cannot enter in edition mode for any connector
> because there is a problem with Velocity.
> We obtain the following error in the logs:
>
> {code:java}
> java.lang.NoSuchMethodError: 'void
> org.apache.velocity.app.VelocityEngine.setExtendedProperties(org.apache.commons.collections.ExtendedProperties)'
> at
> org.apache.manifoldcf.core.i18n.Messages.createVelocityEngine(Messages.java:62)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.outputResourceWithVelocity(Messages.java:132)
> ~[mcf-ui-core.jar:?]
> at
> com.francelabs.datafari.connectors.share.Messages.outputResourceWithVelocity(Messages.java:111)
> ~[?:?]
> at
> com.francelabs.datafari.connectors.share.SharedDriveConnector.outputSpecificationHeader(SharedDriveConnector.java:2829)
> ~[?:?]
> at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:977)
> ~[mcf-crawler-ui.jar:?]
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> ~[jasper.jar:9.0.56]
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> ~[jasper.jar:9.0.56]
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> ~[tomcat-websocket.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
> [catalina.jar:9.0.56]
> at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
> [tomcat-util.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
> [tomcat-util.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>
[jira] [Commented] (CONNECTORS-1712) Broken Velocity UI
[
https://issues.apache.org/jira/browse/CONNECTORS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533351#comment-17533351
]
Karl Wright commented on CONNECTORS-1712:
-
Looking into why the old dependency is still downloaded. It should not be
unless it was not properly cleaned.
> Broken Velocity UI
> --
>
> Key: CONNECTORS-1712
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1712
> Project: ManifoldCF
> Issue Type: Bug
> Components: API
>Affects Versions: ManifoldCF 2.22
>Reporter: Julien Massiera
>Priority: Critical
>
> In the mcf-crawler-ui, we cannot enter in edition mode for any connector
> because there is a problem with Velocity.
> We obtain the following error in the logs:
>
> {code:java}
> java.lang.NoSuchMethodError: 'void
> org.apache.velocity.app.VelocityEngine.setExtendedProperties(org.apache.commons.collections.ExtendedProperties)'
> at
> org.apache.manifoldcf.core.i18n.Messages.createVelocityEngine(Messages.java:62)
> ~[mcf-core.jar:?]
> at
> org.apache.manifoldcf.ui.i18n.Messages.outputResourceWithVelocity(Messages.java:132)
> ~[mcf-ui-core.jar:?]
> at
> com.francelabs.datafari.connectors.share.Messages.outputResourceWithVelocity(Messages.java:111)
> ~[?:?]
> at
> com.francelabs.datafari.connectors.share.SharedDriveConnector.outputSpecificationHeader(SharedDriveConnector.java:2829)
> ~[?:?]
> at org.apache.jsp.editjob_jsp._jspService(editjob_jsp.java:977)
> ~[mcf-crawler-ui.jar:?]
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> ~[jasper.jar:9.0.56]
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> ~[jasper.jar:9.0.56]
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> ~[jasper.jar:9.0.56]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> ~[servlet-api.jar:4.0.FR]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> ~[tomcat-websocket.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
> ~[catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
> [catalina.jar:9.0.56]
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
> [catalina.jar:9.0.56]
> at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> [tomcat-coyote.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
> [tomcat-util.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
> [tomcat-util.jar:9.0.56]
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> [tomcat-util.jar:9.0.56]
> at java.lang.Thread.run(Thread.java:829) [?:?] {code}
>
>
> After some investigations it seems related to the updated velocity lib
[jira] [Commented] (CONNECTORS-1622) Upgrade to Tika 1.28.x
[ https://issues.apache.org/jira/browse/CONNECTORS-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528999#comment-17528999 ] Karl Wright commented on CONNECTORS-1622: - yes. Resolving. > Upgrade to Tika 1.28.x > -- > > Key: CONNECTORS-1622 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1622 > Project: ManifoldCF > Issue Type: Improvement > Components: Tika extractor >Affects Versions: ManifoldCF 2.13 >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF next > > > Tika has released 1.28.1. Changes can be found from here: > [http://tika.apache.org/1.28.1/|http://tika.apache.org/1.23/] > > please see :[TIKA-3625|https://issues.apache.org/jira/browse/TIKA-3625] and > CONNECTORS-1683 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (CONNECTORS-1622) Upgrade to Tika 1.28.x
[ https://issues.apache.org/jira/browse/CONNECTORS-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1622. - Fix Version/s: ManifoldCF 2.21 (was: ManifoldCF next) Resolution: Fixed > Upgrade to Tika 1.28.x > -- > > Key: CONNECTORS-1622 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1622 > Project: ManifoldCF > Issue Type: Improvement > Components: Tika extractor >Affects Versions: ManifoldCF 2.13 >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.21 > > > Tika has released 1.28.1. Changes can be found from here: > [http://tika.apache.org/1.28.1/|http://tika.apache.org/1.23/] > > please see :[TIKA-3625|https://issues.apache.org/jira/browse/TIKA-3625] and > CONNECTORS-1683 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CONNECTORS-1707) LiveLink Connector Ant build broken
[
https://issues.apache.org/jira/browse/CONNECTORS-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528703#comment-17528703
]
Karl Wright commented on CONNECTORS-1707:
-
Perhaps the problem seen here is simply due to the fact that the version of CXF
has been updated and the people trying this so far have not done
clean-core-deps? I can believe that.
> LiveLink Connector Ant build broken
> ---
>
> Key: CONNECTORS-1707
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1707
> Project: ManifoldCF
> Issue Type: Bug
> Components: LiveLink connector
>Reporter: Piergiorgio Lucidi
>Priority: Major
>
> Trying to build the LiveLink connector executing Ant returns an error. Using
> Maven everything is correctly compiled.
> The cause is related to the WSDL generation, the Ant process is failing but
> it seems to return a success outcome even if we have the following error
> executing ant classcreate-wsdls:
>
> {code:java}
> WSDLToJava Error: org.apache.cxf.bus.extension.ExtensionException: Could not
> load extension class org.apache.cxf.common.util.ASMHelperImpl.{code}
>
> Below the entire output of the Ant build:
>
> {code:java}
> pjlucidi@MBP-Pj csws $ant
> Buildfile:
> /Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/connectors/csws/build.xmlcalculate-condition:precompile-warn:precompile-check:has-RMI-check:compile-interface:jar-interface:has-stubs-check:has-proprietary-materials-check:build-stubs-check:compile-stubs:compile-implementation:setup-rmic:rmic-build-all:compile-rmic:jar-rmistub:lib-rmi:classcreate-wsdls:classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
> [java] SLF4J: Actual binding is of type
> [org.slf4j.impl.SimpleLoggerFactory]
> [java] WARNING: An illegal reflective access operation has occurred
> [java] WARNING: Illegal reflective access by
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> (file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/connector-common-lib/jaxb-impl-2.3.0.jar)
> to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
> [java] WARNING: Please consider reporting this to the maintainers of
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> [java] WARNING: Use --illegal-access=warn to enable warnings of further
> illegal reflective access operations
> [java] WARNING: All illegal access operations will be denied in a future
> release
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding source
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding domsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding staxsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding saxsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding jaxb
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend jaxws
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend jaxws21
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend cxf
> [java] [main] WARN org.apache.velocity.deprecation - configuration key
> 'class.resource.loader.class' has been deprecated in favor of
> 'resource.loader.class.class'
> [java] [main] WARN org.apache.velocity.deprecation - configuration key
> 'resource.loader' has been deprecated in favor of 'resource.loaders'
> [java]
> [java] WSDLToJava Error:
> org.apache.cxf.bus.extension.ExtensionException: Could not load extension
> class org.apache.cxf.common.util.ASMHelperImpl.
> [java]
> [java] Java Result: 1classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
[jira] [Commented] (CONNECTORS-1707) LiveLink Connector Ant build broken
[
https://issues.apache.org/jira/browse/CONNECTORS-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528701#comment-17528701
]
Karl Wright commented on CONNECTORS-1707:
-
[~julienFL], you missed doing ant clean-core-deps then.
This is what you get when you do ant clean-core-deps , then ant make-core-deps:
{code}
C:\wip\mcf\trunk\lib>dir cxf*.jar
Volume in drive C is Windows
Volume Serial Number is F4D8-E4E0
Directory of C:\wip\mcf\trunk\lib
04/27/2022 04:34 AM 1,433,236 cxf-core-3.5.0.jar
04/27/2022 04:33 AM 182,637 cxf-rt-bindings-soap-3.5.0.jar
04/27/2022 04:33 AM38,461 cxf-rt-bindings-xml-3.5.0.jar
04/27/2022 04:33 AM 115,309 cxf-rt-databinding-jaxb-3.5.0.jar
04/27/2022 04:34 AM 704,040 cxf-rt-frontend-jaxrs-3.5.0.jar
04/27/2022 04:33 AM 371,928 cxf-rt-frontend-jaxws-3.5.0.jar
04/27/2022 04:33 AM 107,654 cxf-rt-frontend-simple-3.5.0.jar
04/27/2022 04:34 AM 190,074 cxf-rt-rs-client-3.5.0.jar
04/27/2022 04:34 AM41,604 cxf-rt-security-3.5.0.jar
04/27/2022 04:34 AM 392,007 cxf-rt-transports-http-3.5.0.jar
04/27/2022 04:33 AM62,930 cxf-rt-transports-http-hc-3.5.0.jar
04/27/2022 04:33 AM75,762 cxf-rt-ws-addr-3.5.0.jar
04/27/2022 04:33 AM 215,385 cxf-rt-ws-policy-3.5.0.jar
04/27/2022 04:33 AM 178,390 cxf-rt-wsdl-3.5.0.jar
04/27/2022 04:33 AM 149,625 cxf-tools-common-3.5.0.jar
04/27/2022 04:33 AM74,208 cxf-tools-validator-3.5.0.jar
04/27/2022 04:33 AM65,948 cxf-tools-wsdlto-core-3.5.0.jar
04/27/2022 04:33 AM45,178
cxf-tools-wsdlto-databinding-jaxb-3.5.0.jar
04/27/2022 04:33 AM 172,561 cxf-tools-wsdlto-frontend-jaxws-3.5.0.jar
19 File(s) 4,616,937 bytes
0 Dir(s) 111,555,883,008 bytes free
{code}
> LiveLink Connector Ant build broken
> ---
>
> Key: CONNECTORS-1707
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1707
> Project: ManifoldCF
> Issue Type: Bug
> Components: LiveLink connector
>Reporter: Piergiorgio Lucidi
>Priority: Major
>
> Trying to build the LiveLink connector executing Ant returns an error. Using
> Maven everything is correctly compiled.
> The cause is related to the WSDL generation, the Ant process is failing but
> it seems to return a success outcome even if we have the following error
> executing ant classcreate-wsdls:
>
> {code:java}
> WSDLToJava Error: org.apache.cxf.bus.extension.ExtensionException: Could not
> load extension class org.apache.cxf.common.util.ASMHelperImpl.{code}
>
> Below the entire output of the Ant build:
>
> {code:java}
> pjlucidi@MBP-Pj csws $ant
> Buildfile:
> /Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/connectors/csws/build.xmlcalculate-condition:precompile-warn:precompile-check:has-RMI-check:compile-interface:jar-interface:has-stubs-check:has-proprietary-materials-check:build-stubs-check:compile-stubs:compile-implementation:setup-rmic:rmic-build-all:compile-rmic:jar-rmistub:lib-rmi:classcreate-wsdls:classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
> [java] SLF4J: Actual binding is of type
> [org.slf4j.impl.SimpleLoggerFactory]
> [java] WARNING: An illegal reflective access operation has occurred
> [java] WARNING: Illegal reflective access by
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> (file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/connector-common-lib/jaxb-impl-2.3.0.jar)
> to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
> [java] WARNING: Please consider reporting this to the maintainers of
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> [java] WARNING: Use --illegal-access=warn to enable warnings of further
> illegal reflective access operations
> [java] WARNING: All illegal access operations will be denied in a future
> release
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding source
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding domsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding staxsource
> [java] [main] INFO
[jira] [Commented] (CONNECTORS-1707) LiveLink Connector Ant build broken
[
https://issues.apache.org/jira/browse/CONNECTORS-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528652#comment-17528652
]
Karl Wright commented on CONNECTORS-1707:
-
If you want to try experimenting with the ant method that invokes WSDL-to-Java,
here is the ant production you need to modify (from connector-build.xml):
{code}
http://localhost/${wsdlname}"/>
{code}
Note that it contains references needed for Java 11, as well as ALL the cxf
jars in the classpath.
The connector-build.xml jar lives in framework/buildfiles. It is copied to the
dist area when the framework part of MCF is built, before the connectors get
built, so be sure you modify the copy that is being used by the connector build.
As I said before, I cannot much help here since there is no failure for me.
> LiveLink Connector Ant build broken
> ---
>
> Key: CONNECTORS-1707
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1707
> Project: ManifoldCF
> Issue Type: Bug
> Components: LiveLink connector
>Reporter: Piergiorgio Lucidi
>Priority: Major
>
> Trying to build the LiveLink connector executing Ant returns an error. Using
> Maven everything is correctly compiled.
> The cause is related to the WSDL generation, the Ant process is failing but
> it seems to return a success outcome even if we have the following error
> executing ant classcreate-wsdls:
>
> {code:java}
> WSDLToJava Error: org.apache.cxf.bus.extension.ExtensionException: Could not
> load extension class org.apache.cxf.common.util.ASMHelperImpl.{code}
>
> Below the entire output of the Ant build:
>
> {code:java}
> pjlucidi@MBP-Pj csws $ant
> Buildfile:
> /Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/connectors/csws/build.xmlcalculate-condition:precompile-warn:precompile-check:has-RMI-check:compile-interface:jar-interface:has-stubs-check:has-proprietary-materials-check:build-stubs-check:compile-stubs:compile-implementation:setup-rmic:rmic-build-all:compile-rmic:jar-rmistub:lib-rmi:classcreate-wsdls:classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
> [java] SLF4J: Actual binding is of type
> [org.slf4j.impl.SimpleLoggerFactory]
> [java] WARNING: An illegal reflective access operation has occurred
> [java] WARNING: Illegal reflective access by
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> (file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/connector-common-lib/jaxb-impl-2.3.0.jar)
> to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
> [java] WARNING: Please consider reporting this to the maintainers of
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> [java] WARNING: Use --illegal-access=warn to enable warnings of further
> illegal reflective access operations
> [java] WARNING: All illegal access operations will be denied in a future
> release
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding source
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding domsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding staxsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding saxsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default
[jira] [Commented] (CONNECTORS-1707) LiveLink Connector Ant build broken
[
https://issues.apache.org/jira/browse/CONNECTORS-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528648#comment-17528648
]
Karl Wright commented on CONNECTORS-1707:
-
The class in question is in:
Artifact cxf-core
Group org.apache.cxf
Version 3.5.2
> LiveLink Connector Ant build broken
> ---
>
> Key: CONNECTORS-1707
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1707
> Project: ManifoldCF
> Issue Type: Bug
> Components: LiveLink connector
>Reporter: Piergiorgio Lucidi
>Priority: Major
>
> Trying to build the LiveLink connector executing Ant returns an error. Using
> Maven everything is correctly compiled.
> The cause is related to the WSDL generation, the Ant process is failing but
> it seems to return a success outcome even if we have the following error
> executing ant classcreate-wsdls:
>
> {code:java}
> WSDLToJava Error: org.apache.cxf.bus.extension.ExtensionException: Could not
> load extension class org.apache.cxf.common.util.ASMHelperImpl.{code}
>
> Below the entire output of the Ant build:
>
> {code:java}
> pjlucidi@MBP-Pj csws $ant
> Buildfile:
> /Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/connectors/csws/build.xmlcalculate-condition:precompile-warn:precompile-check:has-RMI-check:compile-interface:jar-interface:has-stubs-check:has-proprietary-materials-check:build-stubs-check:compile-stubs:compile-implementation:setup-rmic:rmic-build-all:compile-rmic:jar-rmistub:lib-rmi:classcreate-wsdls:classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
> [java] SLF4J: Actual binding is of type
> [org.slf4j.impl.SimpleLoggerFactory]
> [java] WARNING: An illegal reflective access operation has occurred
> [java] WARNING: Illegal reflective access by
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> (file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/connector-common-lib/jaxb-impl-2.3.0.jar)
> to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
> [java] WARNING: Please consider reporting this to the maintainers of
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> [java] WARNING: Use --illegal-access=warn to enable warnings of further
> illegal reflective access operations
> [java] WARNING: All illegal access operations will be denied in a future
> release
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding source
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding domsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding staxsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding saxsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding jaxb
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend jaxws
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend jaxws21
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend cxf
> [java] [main] WARN org.apache.velocity.deprecation - configuration key
> 'class.resource.loader.class' has been deprecated in favor of
> 'resource.loader.class.class'
> [java] [main] WARN org.apache.velocity.deprecation - configuration key
> 'resource.loader' has been deprecated in favor of 'resource.loaders'
> [java]
> [java] WSDLToJava Error:
> org.apache.cxf.bus.extension.ExtensionException: Could not load extension
> class org.apache.cxf.common.util.ASMHelperImpl.
> [java]
> [java] Java Result: 1classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
>
[jira] [Commented] (CONNECTORS-1707) LiveLink Connector Ant build broken
[
https://issues.apache.org/jira/browse/CONNECTORS-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528645#comment-17528645
]
Karl Wright commented on CONNECTORS-1707:
-
On a windows system running either Java 8 or Java 11, this does not occur.
It sounds like CXF does some kind of autolocate for this class, and it cannot
find it on a Mac. Not sure that we can fix this in ManifoldCF.
> LiveLink Connector Ant build broken
> ---
>
> Key: CONNECTORS-1707
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1707
> Project: ManifoldCF
> Issue Type: Bug
> Components: LiveLink connector
>Reporter: Piergiorgio Lucidi
>Priority: Major
>
> Trying to build the LiveLink connector executing Ant returns an error. Using
> Maven everything is correctly compiled.
> The cause is related to the WSDL generation, the Ant process is failing but
> it seems to return a success outcome even if we have the following error
> executing ant classcreate-wsdls:
>
> {code:java}
> WSDLToJava Error: org.apache.cxf.bus.extension.ExtensionException: Could not
> load extension class org.apache.cxf.common.util.ASMHelperImpl.{code}
>
> Below the entire output of the Ant build:
>
> {code:java}
> pjlucidi@MBP-Pj csws $ant
> Buildfile:
> /Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/connectors/csws/build.xmlcalculate-condition:precompile-warn:precompile-check:has-RMI-check:compile-interface:jar-interface:has-stubs-check:has-proprietary-materials-check:build-stubs-check:compile-stubs:compile-implementation:setup-rmic:rmic-build-all:compile-rmic:jar-rmistub:lib-rmi:classcreate-wsdls:classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
> [java] SLF4J: Actual binding is of type
> [org.slf4j.impl.SimpleLoggerFactory]
> [java] WARNING: An illegal reflective access operation has occurred
> [java] WARNING: Illegal reflective access by
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> (file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/connector-common-lib/jaxb-impl-2.3.0.jar)
> to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
> [java] WARNING: Please consider reporting this to the maintainers of
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> [java] WARNING: Use --illegal-access=warn to enable warnings of further
> illegal reflective access operations
> [java] WARNING: All illegal access operations will be denied in a future
> release
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding source
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding domsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding staxsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding saxsource
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default databinding jaxb
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend jaxws
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend jaxws21
> [java] [main] INFO org.apache.cxf.tools.wsdlto.core.PluginLoader -
> Replaced default frontend cxf
> [java] [main] WARN org.apache.velocity.deprecation - configuration key
> 'class.resource.loader.class' has been deprecated in favor of
> 'resource.loader.class.class'
> [java] [main] WARN org.apache.velocity.deprecation - configuration key
> 'resource.loader' has been deprecated in favor of 'resource.loaders'
> [java]
> [java] WSDLToJava Error:
> org.apache.cxf.bus.extension.ExtensionException: Could not load extension
> class org.apache.cxf.common.util.ASMHelperImpl.
> [java]
> [java] Java Result: 1classcreate-wsdl-cxf:
> [java] SLF4J: Class path contains multiple SLF4J bindings.
> [java] SLF4J: Found binding in
> [jar:file:/Users/pjlucidi/workspaces/manifoldcf/manifoldcf-trunk/dist/lib/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> [java] SLF4J: Found binding in
>
[jira] [Commented] (CONNECTORS-1622) Upgrade to Tika 1.28.x
[
https://issues.apache.org/jira/browse/CONNECTORS-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17496796#comment-17496796
]
Karl Wright commented on CONNECTORS-1622:
-
This issue is now being worked on. There are several new dependencies we have
to include at the appropriate places:
{code}
[INFO] | +- com.googlecode.plist:dd-plist:jar:1.23:compile
[INFO] | +- org.apache.pdfbox:preflight:jar:2.0.25:compile
[INFO] | +- org.apache.pdfbox:xmpbox:jar:2.0.25:compile
[INFO] | | +- org.bouncycastle:bcutil-jdk15on:jar:1.70:compile
[INFO] | | \- com.zaxxer:SparseBitSet:jar:1.2:compile
[INFO] | +- org.tallison:isoparser:jar:1.9.41.7:compile
[INFO] | +- org.tallison:metadata-extractor:jar:2.15.0.1:compile
[INFO] | | \- org.tallison.xmp:xmpcore-shaded:jar:6.1.11:compile
[INFO] | | \- org.apache.cxf:cxf-rt-security:jar:3.5.0:compile
[INFO] | | \- com.zaxxer:HikariCP-java7:jar:2.4.13:compile
[INFO] | | \- com.mchange:mchange-commons-java:jar:0.2.19:compile
[INFO] | | | \- org.apache.sis.core:sis-feature:jar:1.1:compile
{code}
We also have to include mention of these in the licensing text files. This
will likely take a few more days to complete and test.
> Upgrade to Tika 1.28.x
> --
>
> Key: CONNECTORS-1622
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1622
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Tika extractor
>Affects Versions: ManifoldCF 2.13
>Reporter: Cihad Guzel
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF next
>
>
> Tika has released 1.28.1. Changes can be found from here:
> [http://tika.apache.org/1.28.1/|http://tika.apache.org/1.23/]
>
> please see :[TIKA-3625|https://issues.apache.org/jira/browse/TIKA-3625] and
> CONNECTORS-1683
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CONNECTORS-1699) Upgrade to Tika 2.x
[
https://issues.apache.org/jira/browse/CONNECTORS-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17496421#comment-17496421
]
Karl Wright commented on CONNECTORS-1699:
-
Attempting to find the dependencies for 2.3.0 comes up with this error:
{code}
[ERROR] Failed to execute goal on project mcf-tika-connector: Could not resolve
dependencies for project
org.apache.manifoldcf:mcf-tika-connector:jar:2.21-SNAPSHOT: Could not find
artifact org.apache.tika:tika-parsers:jar:2.3.0 in central
(https://repo.maven.apache.org/maven2)
{code}
It looks like 2.x repackaged everything. Somebody therefore needs to invest
the basic time to figure out where everything went. A port to 2.x will likely
require substantial work and time and I certainly do not have it right now.
Volunteers welcome.
> Upgrade to Tika 2.x
> ---
>
> Key: CONNECTORS-1699
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1699
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Tika extractor
>Affects Versions: ManifoldCF 2.21
>Reporter: Cihad Guzel
>Priority: Major
> Fix For: ManifoldCF next
>
>
> Tika has a new version as 2.x . We can support the new version instead of 1.x
> . There is a migration document here:
> [https://cwiki.apache.org/confluence/display/TIKA/Migrating+to+Tika+2.0.0]
> Tika has released 2.3.0. Changes can be found from here:
> https://tika.apache.org/2.3.0/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CONNECTORS-1695) Sitemap xml not detected in version 2.17 webconnector
[
https://issues.apache.org/jira/browse/CONNECTORS-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17482114#comment-17482114
]
Karl Wright commented on CONNECTORS-1695:
-
The interestingMimeTypes are mime types that may be HTML or XHTML, or documents
where links cannot be extracted but where the content can be indexed. What
actually happens to a given document depends on what is actually in it rather
than what the mime type says. The mimetypes are a crude filter only.
The code that we'd want to modify would be the extractLinks code:
{code}
// Now, extract links.
// We'll call the "link extractor" series, so we can plug more
stuff in over time.
boolean indexDocument =
extractLinks(documentIdentifier,activities,filter);
{code}
The code for this method is:
{code}
/** Code to extract links from an already-fetched document. */
protected boolean extractLinks(String
documentIdentifier, IProcessActivity activities, DocumentURLFilter filter)
throws ManifoldCFException, ServiceInterruption
{
ProcessActivityRedirectionHandler redirectHandler = new
ProcessActivityRedirectionHandler(documentIdentifier,activities,filter);
handleRedirects(documentIdentifier,redirectHandler);
if (Logging.connectors.isDebugEnabled() && redirectHandler.shouldIndex() ==
false)
Logging.connectors.debug("Web: Not
indexing document '"+documentIdentifier+"' because of redirection");
// For html, we don't want any actions, because we don't do form submission.
ProcessActivityHTMLHandler htmlHandler = new
ProcessActivityHTMLHandler(documentIdentifier,activities,filter,metaRobotsTagsUsage);
handleHTML(documentIdentifier,htmlHandler);
if (Logging.connectors.isDebugEnabled()
&& htmlHandler.shouldIndex() == false)
Logging.connectors.debug("Web: Not indexing document '"+documentIdentifier+"'
because of HTML robots or content tags prohibiting indexing");
ProcessActivityXMLHandler xmlHandler = new
ProcessActivityXMLHandler(documentIdentifier,activities,filter);
handleXML(documentIdentifier,xmlHandler);
if
(Logging.connectors.isDebugEnabled() && xmlHandler.shouldIndex() == false)
Logging.connectors.debug("Web: Not
indexing document '"+documentIdentifier+"' because of XML robots or content
tags prohibiting indexing");
// May add more later for other extraction tasks.
return htmlHandler.shouldIndex() && redirectHandler.shouldIndex() &&
xmlHandler.shouldIndex();
}
{code}
Note that there are three different parsing attempts made: HTML, XML (which is
I believe RSS feeds only at this point) and redirection pages. You could add a
fourth.
Most of these invoke the fuzzyml parser, which is a bottom-up parser with
overrides for specific interesting tags. Even though the sitemap xml is
supposedly well formed, you wouldn't want to bet on it, and the fuzzyml parser
would be a reasonable technology to do parsing of this kind since it is quite
resilient against syntax errors of all kinds.
So the trick would be to identify the tag structure of a sitemap document and
extend the overrides present for the parser the web connector is using to
understand that xml syntax IN ADDITION TO the xhtml it already understands.
> Sitemap xml not detected in version 2.17 webconnector
> -
>
> Key: CONNECTORS-1695
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1695
> Project: ManifoldCF
> Issue Type: Bug
[jira] [Commented] (CONNECTORS-1695) Sitemap xml not detected in version 2.17 webconnector
[ https://issues.apache.org/jira/browse/CONNECTORS-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17481420#comment-17481420 ] Karl Wright commented on CONNECTORS-1695: - The problem is that there is no parser in the web connector for the sitemap xml and text/xml is not sufficient indication as to what kind of XML it is to do anything special with it. The web crawler accepts files of this kind but assumes they are xhtml, which doesn't work because these aren't. As for processing the whole file, well certainly. That is what should happen. The file should be read in and parsed and the documents within queued for further processing. > Sitemap xml not detected in version 2.17 webconnector > - > > Key: CONNECTORS-1695 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1695 > Project: ManifoldCF > Issue Type: Bug > Components: Web connector >Affects Versions: ManifoldCF 2.17 >Reporter: DK >Priority: Major > > Trying to index sitemap xml and web connector index the whole xml into solr. > Please fix in version 2.17. > If it is any special config that needs to be taken care, please add here and > add in documentation to make it clear. > > Sitemap.xml: > http://www.sitemaps.org/schemas/sitemap/0.9;> > > https:///sitemap_1.xml > 2022-01-21T16:04:45Z > > > > sitemap_1.xml: > > > https:// > 2018-10-31T11:25:27Z > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (CONNECTORS-1604) Documentation to run Manifold over SSL
[ https://issues.apache.org/jira/browse/CONNECTORS-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480386#comment-17480386 ] Karl Wright commented on CONNECTORS-1604: - Two points: First, the standard configuration files for jetty are included in all example distributions and are provided to Jetty during startup of the webapps. So you should be able to configure Jetty any way you like. We are not Jetty experts, however, and how you do this changes from Jetty release to release. The best we can do is point at Jetty documentation somewhere. Second, the examples contain webapp files which do not need to be deployed on Jetty but can easily be deployed on Tomcat. If you cannot get anywhere with Jetty you always have that option. > Documentation to run Manifold over SSL > -- > > Key: CONNECTORS-1604 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1604 > Project: ManifoldCF > Issue Type: Improvement > Components: Documentation >Reporter: DK >Priority: Major > > Documentation does not talk about running manifold over SSL anywhere. I am > sure lot of users who deployed manifold in prod configured ssl. I tried to > configure SSL without success. Can someone point me to instructions to run > manifold over SSL please? > Thanks -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (CONNECTORS-1665) WebConnector: Add activity records for excluded URLs
[ https://issues.apache.org/jira/browse/CONNECTORS-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1665: --- Assignee: Julien Massiera > WebConnector: Add activity records for excluded URLs > - > > Key: CONNECTORS-1665 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1665 > Project: ManifoldCF > Issue Type: Improvement > Components: Web connector >Affects Versions: ManifoldCF 2.18 >Reporter: Julien Massiera >Assignee: Julien Massiera >Priority: Trivial > Fix For: ManifoldCF 2.19 > > Attachments: patch-CONNECTORS-1665 > > > It would be interesting to add activity records in the WebConnector to keep > track of excluded URLs that match an exclude filter -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (CONNECTORS-1665) WebConnector: Add activity records for excluded URLs
[ https://issues.apache.org/jira/browse/CONNECTORS-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480238#comment-17480238 ] Karl Wright commented on CONNECTORS-1665: - Please go ahead and commit this. > WebConnector: Add activity records for excluded URLs > - > > Key: CONNECTORS-1665 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1665 > Project: ManifoldCF > Issue Type: Improvement > Components: Web connector >Affects Versions: ManifoldCF 2.18 >Reporter: Julien Massiera >Assignee: Julien Massiera >Priority: Trivial > Fix For: ManifoldCF 2.19 > > Attachments: patch-CONNECTORS-1665 > > > It would be interesting to add activity records in the WebConnector to keep > track of excluded URLs that match an exclude filter -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (CONNECTORS-1680) WebConnector: Support the Document Base URL element
[
https://issues.apache.org/jira/browse/CONNECTORS-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461575#comment-17461575
]
Karl Wright edited comment on CONNECTORS-1680 at 12/17/21, 5:22 PM:
The baseDocumentIdentifier code is new for adding the ability to set the base
URL for all relative references, a change you requested and I implemented.
All relative document urls should be based on baseDocumentIdentifier, and that
is initialized to be documentIdentifier at the start. Since this is obviously
not happening you are correct in your diagnosis.
was (Author: kwri...@metacarta.com):
The baseDocumentIdentifier code is new for adding the ability to set the base
URL for all relative references, a change you requested and I implemented.
All relative document urls should be based on baseDocumentIdentifier, and that
is initialized to be documentIdentifier at the start.
> WebConnector: Support the Document Base URL element
> ---
>
> Key: CONNECTORS-1680
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1680
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Web connector
>Reporter: Markus Schuch
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.21
>
>
> HTML allows to specifiy the base URL to use for all relative URLs in a
> document:
> {code:java}
>
>
>
> https://example.org/"/>
> ...
>
>...
> {code}
> [https://developer.mozilla.org/de/docs/Web/HTML/Element/base]
> The Web Connector should respect this element when handling relative links.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
