[jira] [Commented] (CONNECTORS-1643) Error: Repeated service interruptions - failure processing document: Failed to acquire credits in time
[ https://issues.apache.org/jira/browse/CONNECTORS-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110992#comment-17110992 ] Karl Wright commented on CONNECTORS-1643: - The proper forum for this question is not to open a bug ticket, but to post is the user mailing list, as described here: http://manifoldcf.apache.org/en_US/mail.html The smb exception means that it is coming from the JCIFS library, which is trying to find documents and their metadata from your windows shares, and is apparently not getting something it needs back promptly. Perhaps the user you are using to do the crawl has insufficient privileges? Also, the error you are seeing is a new one; I've never seen that before, so the connector hasn't either, and it basically doesn't know whether to skip the document or hard fail. But what I'd do is try to open the document yourself in Windows and find out whether it seems to work or not, for a start. > Error: Repeated service interruptions - failure processing document: Failed > to acquire credits in time > -- > > Key: CONNECTORS-1643 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1643 > Project: ManifoldCF > Issue Type: Bug > Components: JCIFS connector >Affects Versions: ManifoldCF 2.14 >Reporter: Ritika Jain >Priority: Major > Fix For: ManifoldCF 2.14 > > Original Estimate: 24h > Remaining Estimate: 24h > > Hi All, > > I am configured Units job (Manifoldcf 2.14 and ES 7.6.2 and postgres 9.6.10) > on server to access files from samba SMBv3 server and used jcifs-ng-2.1.2.jar > to be loaded in lib of manifoldcf. > > After ingesting some records into the index , the got this error in logs :- > :-Unrecognized SmbException thrown getting document version for > smb://store1.directory.intra/folders/UnitsTag1/Hydraulic Engineering/13 HYE > Data/morelis/VSS/MatlabTools/data/s/srca.a > jcifs.smb.SmbException: Failed to acquire credits in time. > > Can anybody please help me understand what can be the possible cause of this > error. Can it be a network connection issue or something else. > > For info:- no authority connection/ Active Directory is being used till now. > Also the Use SID for security (checkbox on manifoldcf UI):- is UNCHECKED. > > Any help will be appreciated greatly. > > Thanks > RItika -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CONNECTORS-1643) Error: Repeated service interruptions - failure processing document: Failed to acquire credits in time
[ https://issues.apache.org/jira/browse/CONNECTORS-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1643: --- Assignee: Karl Wright > Error: Repeated service interruptions - failure processing document: Failed > to acquire credits in time > -- > > Key: CONNECTORS-1643 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1643 > Project: ManifoldCF > Issue Type: Bug > Components: JCIFS connector >Affects Versions: ManifoldCF 2.14 >Reporter: Ritika Jain >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.14 > > Original Estimate: 24h > Remaining Estimate: 24h > > Hi All, > > I am configured Units job (Manifoldcf 2.14 and ES 7.6.2 and postgres 9.6.10) > on server to access files from samba SMBv3 server and used jcifs-ng-2.1.2.jar > to be loaded in lib of manifoldcf. > > After ingesting some records into the index , the got this error in logs :- > :-Unrecognized SmbException thrown getting document version for > smb://store1.directory.intra/folders/UnitsTag1/Hydraulic Engineering/13 HYE > Data/morelis/VSS/MatlabTools/data/s/srca.a > jcifs.smb.SmbException: Failed to acquire credits in time. > > Can anybody please help me understand what can be the possible cause of this > error. Can it be a network connection issue or something else. > > For info:- no authority connection/ Active Directory is being used till now. > Also the Use SID for security (checkbox on manifoldcf UI):- is UNCHECKED. > > Any help will be appreciated greatly. > > Thanks > RItika -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CONNECTORS-1642) PostgreSQL Version >= 12.2 DB Initialization Problems
[
https://issues.apache.org/jira/browse/CONNECTORS-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1642.
-
Fix Version/s: ManifoldCF 2.16
Resolution: Fixed
r1877648.
Worked fine on Postgresql 9, so I think we should be good. I would appreciate
it if [~wolfingeru] would try this against Postgresql 12.2 however.
> PostgreSQL Version >= 12.2 DB Initialization Problems
> -
>
> Key: CONNECTORS-1642
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1642
> Project: ManifoldCF
> Issue Type: Bug
> Components: Framework core
>Affects Versions: ManifoldCF 2.15
>Reporter: Uwe Wolfinger
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.16
>
> Attachments: image-2020-05-12-14-34-45-754.png,
> image-2020-05-12-14-35-41-885.png
>
>
> when trying to run the "./executecommand.sh
> org.apache.manifoldcf.crawler.InitializeAndRegister" script, the following
> erro shows up and the initialization process stops:
> {{ WARNING: Illegal reflective access by org.postgresql.jdbc.TimestampUtils
> ([file:/home/suche/crawler/lib/postgresql-42.1.3.jar|file:///home/suche/crawler/lib/postgresql-42.1.3.jar])
> to field java.util.TimeZone.defaultTimeZone}}
> {{ WARNING: Please consider reporting this to the maintainers of
> org.postgresql.jdbc.TimestampUtils}}
> {{ WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations}}
> {{ WARNING: All illegal access operations will be denied in a future release}}
> {{ org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
> exception: SQLException doing query (42703): FEHLER: Spalte pg_attrdef.adsrc
> existiert nicht}}
> {{ Position: 447}}
> {{ at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.finishUp(Database.java:715)}}
> {{ at
> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:741)}}
> {{ at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:803)}}
> {{ at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1457)}}
> {{ at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146)}}
> {{ at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)}}
> {{ at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:837)}}
> {{ at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.getTableSchema(DBInterfacePostgreSQL.java:696)}}
> {{ at
> org.apache.manifoldcf.core.database.BaseTable.getTableSchema(BaseTable.java:185)}}
> {{ at
> org.apache.manifoldcf.agents.agentmanager.AgentManager.install(AgentManager.java:67)}}
> {{ at
> org.apache.manifoldcf.agents.system.ManifoldCF.installTables(ManifoldCF.java:112)}}
>
> the column "pg_attrdef.adsrc" no longer exists in PostgreSQL DB 12.2.
> [https://www.postgresql.org/docs/11/catalog-pg-attrdef.html]
> which means that it is impossible to initialize the core DB in a PostgreSQL
> 12.2
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1642) PostgreSQL Version >= 12.2 DB Initialization Problems
[
https://issues.apache.org/jira/browse/CONNECTORS-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105378#comment-17105378
]
Karl Wright commented on CONNECTORS-1642:
-
[~michaelcizmar] Yes, a "latest versions" issue.
[~wolfingeru] Do your admins think this change will work back to PGSQL 9 or so?
I am not much worried about supporting versions older than that.
> PostgreSQL Version >= 12.2 DB Initialization Problems
> -
>
> Key: CONNECTORS-1642
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1642
> Project: ManifoldCF
> Issue Type: Bug
> Components: Framework core
>Affects Versions: ManifoldCF 2.15
>Reporter: Uwe Wolfinger
>Assignee: Karl Wright
>Priority: Major
>
> when trying to run the "./executecommand.sh
> org.apache.manifoldcf.crawler.InitializeAndRegister" script, the following
> erro shows up and the initialization process stops:
> {{ WARNING: Illegal reflective access by org.postgresql.jdbc.TimestampUtils
> ([file:/home/suche/crawler/lib/postgresql-42.1.3.jar|file:///home/suche/crawler/lib/postgresql-42.1.3.jar])
> to field java.util.TimeZone.defaultTimeZone}}
> {{ WARNING: Please consider reporting this to the maintainers of
> org.postgresql.jdbc.TimestampUtils}}
> {{ WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations}}
> {{ WARNING: All illegal access operations will be denied in a future release}}
> {{ org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
> exception: SQLException doing query (42703): FEHLER: Spalte pg_attrdef.adsrc
> existiert nicht}}
> {{ Position: 447}}
> {{ at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.finishUp(Database.java:715)}}
> {{ at
> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:741)}}
> {{ at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:803)}}
> {{ at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1457)}}
> {{ at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146)}}
> {{ at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)}}
> {{ at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:837)}}
> {{ at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.getTableSchema(DBInterfacePostgreSQL.java:696)}}
> {{ at
> org.apache.manifoldcf.core.database.BaseTable.getTableSchema(BaseTable.java:185)}}
> {{ at
> org.apache.manifoldcf.agents.agentmanager.AgentManager.install(AgentManager.java:67)}}
> {{ at
> org.apache.manifoldcf.agents.system.ManifoldCF.installTables(ManifoldCF.java:112)}}
>
> the column "pg_attrdef.adsrc" no longer exists in PostgreSQL DB 12.2.
> [https://www.postgresql.org/docs/11/catalog-pg-attrdef.html]
> which means that it is impossible to initialize the core DB in a PostgreSQL
> 12.2
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1642) PostgreSQL Version >= 12.2 DB Initialization Problems
[
https://issues.apache.org/jira/browse/CONNECTORS-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105367#comment-17105367
]
Karl Wright commented on CONNECTORS-1642:
-
The full code of the current method is the following:
{code}
/** Get a table's schema.
*@param tableName is the name of the table.
*@param cacheKeys are the keys against which to cache the query, or null.
*@param queryClass is the name of the query class, or null.
*@return a map of column names and ColumnDescription objects, describing the
schema, or null if the
* table doesn't exist.
*/
@Override
public Map getTableSchema(String tableName,
StringSet cacheKeys, String queryClass)
throws ManifoldCFException
{
StringBuilder query = new StringBuilder();
List list = new ArrayList();
query.append("SELECT pg_attribute.attname AS \"Field\",");
query.append("CASE pg_type.typname WHEN 'int2' THEN 'smallint' WHEN 'int4'
THEN 'int'");
query.append(" WHEN 'int8' THEN 'bigint' WHEN 'varchar' THEN 'varchar(' ||
pg_attribute.atttypmod-4 || ')'");
query.append(" WHEN 'text' THEN 'longtext'");
query.append(" WHEN 'bpchar' THEN 'char(' || pg_attribute.atttypmod-4 ||
')'");
query.append(" ELSE pg_type.typname END AS \"Type\",");
query.append("CASE WHEN pg_attribute.attnotnull THEN '' ELSE 'YES' END AS
\"Null\",");
query.append("CASE pg_type.typname WHEN 'varchar' THEN
substring(pg_attrdef.adsrc from '^(.*).*$') ELSE pg_attrdef.adsrc END AS
Default ");
query.append("FROM pg_class INNER JOIN pg_attribute ON
(pg_class.oid=pg_attribute.attrelid) INNER JOIN pg_type ON
(pg_attribute.atttypid=pg_type.oid) ");
query.append("LEFT JOIN pg_attrdef ON (pg_class.oid=pg_attrdef.adrelid AND
pg_attribute.attnum=pg_attrdef.adnum) ");
query.append("WHERE pg_class.relname=? AND pg_attribute.attnum>=1 AND NOT
pg_attribute.attisdropped ");
query.append("ORDER BY pg_attribute.attnum");
list.add(tableName);
IResultSet set = performQuery(query.toString(),list,cacheKeys,queryClass);
if (set.getRowCount() == 0)
return null;
// Digest the result
Map rval = new
HashMap();
int i = 0;
while (i < set.getRowCount())
{
IResultRow row = set.getRow(i++);
String fieldName = row.getValue("Field").toString();
String type = row.getValue("Type").toString();
boolean isNull = row.getValue("Null").toString().equals("YES");
boolean isPrimaryKey = false; //
row.getValue("Key").toString().equals("PRI");
rval.put(fieldName,new
ColumnDescription(type,isPrimaryKey,isNull,null,null,false));
}
return rval;
}
{code}
The query I got from StackOverflow or the Postgresql manual (can't remember
which) a decade ago. I think just replacing the query with a more modern
version would work but I have no idea what the more modern version would be.
Patches welcome.
> PostgreSQL Version >= 12.2 DB Initialization Problems
> -
>
> Key: CONNECTORS-1642
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1642
> Project: ManifoldCF
> Issue Type: Bug
> Components: Framework core
>Affects Versions: ManifoldCF 2.15
>Reporter: Uwe Wolfinger
>Assignee: Karl Wright
>Priority: Major
>
> when trying to run the "./executecommand.sh
> org.apache.manifoldcf.crawler.InitializeAndRegister" script, the following
> erro shows up and the initialization process stops:
> {{ WARNING: Illegal reflective access by org.postgresql.jdbc.TimestampUtils
> ([file:/home/suche/crawler/lib/postgresql-42.1.3.jar|file:///home/suche/crawler/lib/postgresql-42.1.3.jar])
> to field java.util.TimeZone.defaultTimeZone}}
> {{ WARNING: Please consider reporting this to the maintainers of
> org.postgresql.jdbc.TimestampUtils}}
> {{ WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations}}
> {{ WARNING: All illegal access operations will be denied in a future release}}
> {{ org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
> exception: SQLException doing query (42703): FEHLER: Spalte pg_attrdef.adsrc
> existiert nicht}}
> {{ Position: 447}}
> {{ at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.finishUp(Database.java:715)}}
> {{ at
> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:741)}}
> {{ at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:803)}}
> {{ at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1457)}}
> {{ at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146)}}
> {{ at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)}}
> {{ at
>
[jira] [Commented] (CONNECTORS-1642) PostgreSQL Version >= 12.2 DB Initialization Problems
[
https://issues.apache.org/jira/browse/CONNECTORS-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105355#comment-17105355
]
Karl Wright commented on CONNECTORS-1642:
-
Sounds like the method DBInterfacePostgreSQL.getTableSchema() needs to be
updated to reflect the newest Postgresql version. Does anyone want to propose
a patch for this?
> PostgreSQL Version >= 12.2 DB Initialization Problems
> -
>
> Key: CONNECTORS-1642
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1642
> Project: ManifoldCF
> Issue Type: Bug
> Components: Framework core
>Affects Versions: ManifoldCF 2.15
>Reporter: Uwe Wolfinger
>Assignee: Karl Wright
>Priority: Major
>
> when trying to run the "./executecommand.sh
> org.apache.manifoldcf.crawler.InitializeAndRegister" script, the following
> erro shows up and the initialization process stops:
> {{ WARNING: Illegal reflective access by org.postgresql.jdbc.TimestampUtils
> ([file:/home/suche/crawler/lib/postgresql-42.1.3.jar|file:///home/suche/crawler/lib/postgresql-42.1.3.jar])
> to field java.util.TimeZone.defaultTimeZone}}
> {{ WARNING: Please consider reporting this to the maintainers of
> org.postgresql.jdbc.TimestampUtils}}
> {{ WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations}}
> {{ WARNING: All illegal access operations will be denied in a future release}}
> {{ org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
> exception: SQLException doing query (42703): FEHLER: Spalte pg_attrdef.adsrc
> existiert nicht}}
> {{ Position: 447}}
> {{ at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.finishUp(Database.java:715)}}
> {{ at
> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:741)}}
> {{ at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:803)}}
> {{ at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1457)}}
> {{ at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146)}}
> {{ at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)}}
> {{ at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:837)}}
> {{ at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.getTableSchema(DBInterfacePostgreSQL.java:696)}}
> {{ at
> org.apache.manifoldcf.core.database.BaseTable.getTableSchema(BaseTable.java:185)}}
> {{ at
> org.apache.manifoldcf.agents.agentmanager.AgentManager.install(AgentManager.java:67)}}
> {{ at
> org.apache.manifoldcf.agents.system.ManifoldCF.installTables(ManifoldCF.java:112)}}
>
> the column "pg_attrdef.adsrc" no longer exists in PostgreSQL DB 12.2.
> [https://www.postgresql.org/docs/11/catalog-pg-attrdef.html]
> which means that it is impossible to initialize the core DB in a PostgreSQL
> 12.2
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CONNECTORS-1642) PostgreSQL Version >= 12.2 DB Initialization Problems
[
https://issues.apache.org/jira/browse/CONNECTORS-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1642:
---
Assignee: Karl Wright
> PostgreSQL Version >= 12.2 DB Initialization Problems
> -
>
> Key: CONNECTORS-1642
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1642
> Project: ManifoldCF
> Issue Type: Bug
> Components: Framework core
>Affects Versions: ManifoldCF 2.15
>Reporter: Uwe Wolfinger
>Assignee: Karl Wright
>Priority: Major
>
> when trying to run the "./executecommand.sh
> org.apache.manifoldcf.crawler.InitializeAndRegister" script, the following
> erro shows up and the initialization process stops:
> {{ WARNING: Illegal reflective access by org.postgresql.jdbc.TimestampUtils
> ([file:/home/suche/crawler/lib/postgresql-42.1.3.jar|file:///home/suche/crawler/lib/postgresql-42.1.3.jar])
> to field java.util.TimeZone.defaultTimeZone}}
> {{ WARNING: Please consider reporting this to the maintainers of
> org.postgresql.jdbc.TimestampUtils}}
> {{ WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations}}
> {{ WARNING: All illegal access operations will be denied in a future release}}
> {{ org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
> exception: SQLException doing query (42703): FEHLER: Spalte pg_attrdef.adsrc
> existiert nicht}}
> {{ Position: 447}}
> {{ at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.finishUp(Database.java:715)}}
> {{ at
> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:741)}}
> {{ at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:803)}}
> {{ at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1457)}}
> {{ at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146)}}
> {{ at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)}}
> {{ at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:837)}}
> {{ at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.getTableSchema(DBInterfacePostgreSQL.java:696)}}
> {{ at
> org.apache.manifoldcf.core.database.BaseTable.getTableSchema(BaseTable.java:185)}}
> {{ at
> org.apache.manifoldcf.agents.agentmanager.AgentManager.install(AgentManager.java:67)}}
> {{ at
> org.apache.manifoldcf.agents.system.ManifoldCF.installTables(ManifoldCF.java:112)}}
>
> the column "pg_attrdef.adsrc" no longer exists in PostgreSQL DB 12.2.
> [https://www.postgresql.org/docs/11/catalog-pg-attrdef.html]
> which means that it is impossible to initialize the core DB in a PostgreSQL
> 12.2
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1641) Cannot find doap file: http://manifoldcf.apache.org/doap_ManifoldCF.rdf
[
https://issues.apache.org/jira/browse/CONNECTORS-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103243#comment-17103243
]
Karl Wright commented on CONNECTORS-1641:
-
It is present in SVN:
{code}
C:\wip\mcf-site\publish>dir
Volume in drive C is Windows
Volume Serial Number is F4D8-E4E0
Directory of C:\wip\mcf-site\publish
05/08/2020 08:22 AM .
05/08/2020 08:22 AM ..
05/08/2020 08:22 AM 3,845 .htaccess
05/08/2020 08:22 AM33 broken-links.xml
05/08/2020 08:22 AM 3,013 doap_ManifoldCF.rdf
05/08/2020 08:22 AM en_US
05/08/2020 08:22 AM images
05/08/2020 08:22 AM 5,081 index.html
05/08/2020 08:22 AM ja_JP
05/08/2020 08:22 AM16,599 linkmap.html
05/08/2020 08:20 AM release
05/08/2020 08:38 AM skin
05/08/2020 08:38 AM zh_CN
5 File(s) 28,571 bytes
8 Dir(s) 123,887,923,200 bytes free
C:\wip\mcf-site\publish>
{code}
and here:
{code}
C:\wip\mcf-site\publish>svn list
https://svn.apache.org/repos/asf/manifoldcf/site/publish
.htaccess
broken-links.xml
doap_ManifoldCF.rdf
en_US/
images/
index.html
ja_JP/
linkmap.html
release/
skin/
zh_CN/
{code}
I guess this needs to be escalated to INFRA.
> Cannot find doap file: http://manifoldcf.apache.org/doap_ManifoldCF.rdf
> ---
>
> Key: CONNECTORS-1641
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1641
> Project: ManifoldCF
> Issue Type: Bug
>Reporter: Sebb
>Priority: Major
>
> URL: http://manifoldcf.apache.org/doap_ManifoldCF.rdf
> HTTP Error 404: Not Found
> Source:
> https://svn.apache.org/repos/asf/comdev/projects.apache.org/trunk/data/projects.xml
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CONNECTORS-1640) http://manifoldcf.apache.org/ Website missing
[ https://issues.apache.org/jira/browse/CONNECTORS-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1640. - Fix Version/s: ManifoldCF 2.16 Resolution: Fixed Looks like it finally went live. > http://manifoldcf.apache.org/ Website missing > - > > Key: CONNECTORS-1640 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1640 > Project: ManifoldCF > Issue Type: Bug > Environment: http://manifoldcf.apache.org/ >Reporter: Sebb >Assignee: Karl Wright >Priority: Critical > Fix For: ManifoldCF 2.16 > > > As the subject says - there is currently no usable website at > http://manifoldcf.apache.org/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1640) http://manifoldcf.apache.org/ Website missing
[ https://issues.apache.org/jira/browse/CONNECTORS-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102566#comment-17102566 ] Karl Wright commented on CONNECTORS-1640: - I've re-committed to the svn instance that the site is mirrored from. The workarea looks fine. But the site still doesn't to appear to have changed, so I may need to involve Infra. > http://manifoldcf.apache.org/ Website missing > - > > Key: CONNECTORS-1640 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1640 > Project: ManifoldCF > Issue Type: Bug > Environment: http://manifoldcf.apache.org/ >Reporter: Sebb >Assignee: Karl Wright >Priority: Critical > > As the subject says - there is currently no usable website at > http://manifoldcf.apache.org/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1640) http://manifoldcf.apache.org/ Website missing
[ https://issues.apache.org/jira/browse/CONNECTORS-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102528#comment-17102528 ] Karl Wright commented on CONNECTORS-1640: - Looks like svn failed spectacularly for some reason here. Retrying the publish now. > http://manifoldcf.apache.org/ Website missing > - > > Key: CONNECTORS-1640 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1640 > Project: ManifoldCF > Issue Type: Bug > Environment: http://manifoldcf.apache.org/ >Reporter: Sebb >Priority: Critical > > As the subject says - there is currently no usable website at > http://manifoldcf.apache.org/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CONNECTORS-1640) http://manifoldcf.apache.org/ Website missing
[ https://issues.apache.org/jira/browse/CONNECTORS-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1640: --- Assignee: Karl Wright > http://manifoldcf.apache.org/ Website missing > - > > Key: CONNECTORS-1640 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1640 > Project: ManifoldCF > Issue Type: Bug > Environment: http://manifoldcf.apache.org/ >Reporter: Sebb >Assignee: Karl Wright >Priority: Critical > > As the subject says - there is currently no usable website at > http://manifoldcf.apache.org/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CONNECTORS-1624) Get ManifoldCF to run under Java 11 or higher
[ https://issues.apache.org/jira/browse/CONNECTORS-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1624. - Resolution: Fixed Commits were made to address Java 11 changes, including the ES testing version. > Get ManifoldCF to run under Java 11 or higher > - > > Key: CONNECTORS-1624 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1624 > Project: ManifoldCF > Issue Type: Task > Components: Framework core >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > > Java 11 doesn't include a number of classes that Java 8 does. We need to > explicitly include jars that provide these classes or ManifoldCF will not > function under higher Java revs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CONNECTORS-1639) Upgrade Elastic Search Version
[ https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1639. - Resolution: Fixed Merged branch into trunk. r1877321 > Upgrade Elastic Search Version > -- > > Key: CONNECTORS-1639 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1639 > Project: ManifoldCF > Issue Type: Improvement > Components: Elastic Search connector >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > Attachments: CONNECTORS-1639.diff, > elastic-search-1.0.1-java11-build-error.log, es_start.sh, es_stop.sh > > > Current Elastic Search version is 1.0.1 . According to [this > matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not > supported by any ES version below 6.5. > Besides, ES 1.x is no longer supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1639) Upgrade Elastic Search Version
[ https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097744#comment-17097744 ] Karl Wright commented on CONNECTORS-1639: - Thanks, these are helpful. I will be able to integrate them into the java code for starting and stopping the ES instance. There's only one thing I still need: a compatible ES version and mapper attachment. > Upgrade Elastic Search Version > -- > > Key: CONNECTORS-1639 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1639 > Project: ManifoldCF > Issue Type: Improvement > Components: Elastic Search connector >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > Attachments: CONNECTORS-1639.diff, > elastic-search-1.0.1-java11-build-error.log > > > Current Elastic Search version is 1.0.1 . According to [this > matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not > supported by any ES version below 6.5. > Besides, ES 1.x is no longer supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CONNECTORS-1639) Upgrade Elastic Search Version
[
https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087245#comment-17087245
]
Karl Wright edited comment on CONNECTORS-1639 at 4/19/20, 11:09 PM:
Since this is firing off a java task, we could use the Ant Java task, described
here:
https://ant.apache.org/manual/Tasks/java.html
However, this will not work for maven so I'd steer away from that.
We can do something similar to this instead, which would invoke the same Java
again, although we'd want ES's classpath obviously:
{code}
public final class JavaProcess {
private JavaProcess() {}
public static int exec(Class klass, List args) throws IOException,
InterruptedException {
String javaHome = System.getProperty("java.home");
String javaBin = javaHome +
File.separator + "bin" +
File.separator + "java";
String classpath = System.getProperty("java.class.path");
String className = klass.getName();
List command = new LinkedList();
command.add(javaBin);
command.add("-cp");
command.add(classpath);
command.add(className);
if (args != null) {
command.addAll(args);
}
ProcessBuilder builder = new ProcessBuilder(command);
Process process = builder.inheritIO().start();
process.waitFor();
return process.exitValue();
}
}
{code}
There are, however, the following problems to be addressed: (1) waiting for the
instance to start, and (2) stopping the instance.
was (Author: kwri...@metacarta.com):
Since this is firing off a java task, we could use the Ant Java task, described
here:
https://ant.apache.org/manual/Tasks/java.html
However, this will not work for maven so I'd steer away from that.
We can do something similar to this instead, which would invoke the same Java
again, although we'd want ES's classpath obviously:
{code}
> Upgrade Elastic Search Version
> --
>
> Key: CONNECTORS-1639
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1639
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Elastic Search connector
>Reporter: Cihad Guzel
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.16
>
> Attachments: CONNECTORS-1639.diff,
> elastic-search-1.0.1-java11-build-error.log
>
>
> Current Elastic Search version is 1.0.1 . According to [this
> matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not
> supported by any ES version below 6.5.
> Besides, ES 1.x is no longer supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1639) Upgrade Elastic Search Version
[
https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087245#comment-17087245
]
Karl Wright commented on CONNECTORS-1639:
-
Since this is firing off a java task, we could use the Ant Java task, described
here:
https://ant.apache.org/manual/Tasks/java.html
However, this will not work for maven so I'd steer away from that.
We can do something similar to this instead, which would invoke the same Java
again, although we'd want ES's classpath obviously:
{code}
> Upgrade Elastic Search Version
> --
>
> Key: CONNECTORS-1639
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1639
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Elastic Search connector
>Reporter: Cihad Guzel
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.16
>
> Attachments: CONNECTORS-1639.diff,
> elastic-search-1.0.1-java11-build-error.log
>
>
> Current Elastic Search version is 1.0.1 . According to [this
> matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not
> supported by any ES version below 6.5.
> Besides, ES 1.x is no longer supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1639) Upgrade Elastic Search Version
[
https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086994#comment-17086994
]
Karl Wright commented on CONNECTORS-1639:
-
It looks like we can reverse engineer the startup script here:
{code}
test-materials/elasticsearch-7.6.2/bin/elasticsearch"
{code}
The basic daemon runner is:
{code}
exec \
"$JAVA" \
$ES_JAVA_OPTS \
-Des.path.home="$ES_HOME" \
-Des.path.conf="$ES_PATH_CONF" \
-Des.distribution.flavor="$ES_DISTRIBUTION_FLAVOR" \
-Des.distribution.type="$ES_DISTRIBUTION_TYPE" \
-Des.bundled_jdk="$ES_BUNDLED_JDK" \
-cp "$ES_CLASSPATH" \
org.elasticsearch.bootstrap.Elasticsearch \
"$@"
{code}
This can be shelled out easily enough in Java, BUT we'd also need to make sure
it comes up before starting the test.
> Upgrade Elastic Search Version
> --
>
> Key: CONNECTORS-1639
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1639
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Elastic Search connector
>Reporter: Cihad Guzel
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.16
>
> Attachments: CONNECTORS-1639.diff,
> elastic-search-1.0.1-java11-build-error.log
>
>
> Current Elastic Search version is 1.0.1 . According to [this
> matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not
> supported by any ES version below 6.5.
> Besides, ES 1.x is no longer supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CONNECTORS-1639) Upgrade Elastic Search Version
[
https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086926#comment-17086926
]
Karl Wright edited comment on CONNECTORS-1639 at 4/19/20, 11:31 AM:
Unfortunately the dependencies for ElasticSearch Cluster Runner are huge. See:
https://mvnrepository.com/artifact/org.codelibs/elasticsearch-cluster-runner/7.6.2.0
This is a problem because we'd need to download all these dependencies and
their dependencies. But it may be that the cluster runner does not actually
use all these.
It may not even be the right thing to use. It doesn't seem like it would be
hard to write something that starts up a cluster based on a specific image.
[~michaelcizmar], if you were starting a cluster from a downloaded, unpacked
image, what steps would you take?
FWIW, if you want to try this, svn checkout
https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1639, and then
do: the following:
{code}
ant make-core-deps
ant build
cd connectors/elasticsearch
ant download-dependencies
{code}
The unpacked ES download, and the mapper attachments plugin, should live in
test-materials at that point.
was (Author: kwri...@metacarta.com):
Unfortunately the dependencies for ElasticSearch Cluster Runner are huge. See:
https://mvnrepository.com/artifact/org.codelibs/elasticsearch-cluster-runner/7.6.2.0
This is a problem because we'd need to download all these dependencies and
their dependencies. But it may be that the cluster runner does not actually
use all these.
It may not even be the right thing to use. It doesn't seem like it would be
hard to write something that starts up a cluster based on a specific image.
[~michaelcizmar], if you were starting a cluster from a downloaded, unpacked
image, what steps would you take?
(FWIW, if you want to try this, svn checkout
https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1639, and then
do: the following:
{code}
ant make-core-deps
ant build
cd connectors/elasticsearch
ant download-dependencies
{code}
The unpacked ES download, and the mapper attachments plugin, should live in
test-materials at that point.
> Upgrade Elastic Search Version
> --
>
> Key: CONNECTORS-1639
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1639
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Elastic Search connector
>Reporter: Cihad Guzel
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.16
>
> Attachments: CONNECTORS-1639.diff,
> elastic-search-1.0.1-java11-build-error.log
>
>
> Current Elastic Search version is 1.0.1 . According to [this
> matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not
> supported by any ES version below 6.5.
> Besides, ES 1.x is no longer supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CONNECTORS-1639) Upgrade Elastic Search Version
[
https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086926#comment-17086926
]
Karl Wright edited comment on CONNECTORS-1639 at 4/19/20, 11:31 AM:
Unfortunately the dependencies for ElasticSearch Cluster Runner are huge. See:
https://mvnrepository.com/artifact/org.codelibs/elasticsearch-cluster-runner/7.6.2.0
This is a problem because we'd need to download all these dependencies and
their dependencies. But it may be that the cluster runner does not actually
use all these.
It may not even be the right thing to use. It doesn't seem like it would be
hard to write something that starts up a cluster based on a specific image.
[~michaelcizmar], if you were starting a cluster from a downloaded, unpacked
image, what steps would you take?
(FWIW, if you want to try this, svn checkout
https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1639, and then
do: the following:
{code}
ant make-core-deps
ant build
cd connectors/elasticsearch
ant download-dependencies
{code}
The unpacked ES download, and the mapper attachments plugin, should live in
test-materials at that point.
was (Author: kwri...@metacarta.com):
Unfortunately the dependencies for ElasticSearch Cluster Runner are huge. See:
https://mvnrepository.com/artifact/org.codelibs/elasticsearch-cluster-runner/7.6.2.0
> Upgrade Elastic Search Version
> --
>
> Key: CONNECTORS-1639
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1639
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Elastic Search connector
>Reporter: Cihad Guzel
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.16
>
> Attachments: CONNECTORS-1639.diff,
> elastic-search-1.0.1-java11-build-error.log
>
>
> Current Elastic Search version is 1.0.1 . According to [this
> matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not
> supported by any ES version below 6.5.
> Besides, ES 1.x is no longer supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1639) Upgrade Elastic Search Version
[ https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086926#comment-17086926 ] Karl Wright commented on CONNECTORS-1639: - Unfortunately the dependencies for ElasticSearch Cluster Runner are huge. See: https://mvnrepository.com/artifact/org.codelibs/elasticsearch-cluster-runner/7.6.2.0 > Upgrade Elastic Search Version > -- > > Key: CONNECTORS-1639 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1639 > Project: ManifoldCF > Issue Type: Improvement > Components: Elastic Search connector >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > Attachments: CONNECTORS-1639.diff, > elastic-search-1.0.1-java11-build-error.log > > > Current Elastic Search version is 1.0.1 . According to [this > matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not > supported by any ES version below 6.5. > Besides, ES 1.x is no longer supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1639) Upgrade Elastic Search Version
[ https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086902#comment-17086902 ] Karl Wright commented on CONNECTORS-1639: - Here are some examples of how to invoke these from code: https://www.programcreek.com/java-api-examples/?api=org.codelibs.elasticsearch.runner.ElasticsearchClusterRunner > Upgrade Elastic Search Version > -- > > Key: CONNECTORS-1639 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1639 > Project: ManifoldCF > Issue Type: Improvement > Components: Elastic Search connector >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > Attachments: CONNECTORS-1639.diff, > elastic-search-1.0.1-java11-build-error.log > > > Current Elastic Search version is 1.0.1 . According to [this > matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not > supported by any ES version below 6.5. > Besides, ES 1.x is no longer supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1639) Upgrade Elastic Search Version
[ https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086901#comment-17086901 ] Karl Wright commented on CONNECTORS-1639: - I created a branch (https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1639) that updates the ant build to download ES 7.6.2 and mapper attachments version 3.1.2, and unpacks them. I hope these are compatible? But in any case, all that should be needed now is incorporating Elasticsearch Runner to start the instance programmatically, if that is how it works. Looking into that now. > Upgrade Elastic Search Version > -- > > Key: CONNECTORS-1639 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1639 > Project: ManifoldCF > Issue Type: Improvement > Components: Elastic Search connector >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > Attachments: CONNECTORS-1639.diff, > elastic-search-1.0.1-java11-build-error.log > > > Current Elastic Search version is 1.0.1 . According to [this > matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not > supported by any ES version below 6.5. > Besides, ES 1.x is no longer supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1639) Upgrade Elastic Search Version
[ https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086693#comment-17086693 ] Karl Wright commented on CONNECTORS-1639: - The pom diff you included looks fine. Once you've got a diff for the test itself I can make the rest of the ant changes. > Upgrade Elastic Search Version > -- > > Key: CONNECTORS-1639 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1639 > Project: ManifoldCF > Issue Type: Improvement > Components: Elastic Search connector >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > Attachments: CONNECTORS-1639.diff, > elastic-search-1.0.1-java11-build-error.log > > > Current Elastic Search version is 1.0.1 . According to [this > matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not > supported by any ES version below 6.5. > Besides, ES 1.x is no longer supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1639) Upgrade Elastic Search Version
[ https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086674#comment-17086674 ] Karl Wright commented on CONNECTORS-1639: - [~michaelcizmar] Cihad Guzel has made some commits to get MCF JDK 11 compatible. The elasticsearch maven plugin I can't find in the maven repo, but looking for it I found this: https://mvnrepository.com/artifact/org.codelibs/elasticsearch-cluster-runner It seems like this could in theory be used by our standard integration tests to start a cluster and tear it down, no? > Upgrade Elastic Search Version > -- > > Key: CONNECTORS-1639 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1639 > Project: ManifoldCF > Issue Type: Improvement > Components: Elastic Search connector >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > Attachments: CONNECTORS-1639.diff, > elastic-search-1.0.1-java11-build-error.log > > > Current Elastic Search version is 1.0.1 . According to [this > matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not > supported by any ES version below 6.5. > Besides, ES 1.x is no longer supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CONNECTORS-1639) Upgrade Elastic Search Version
[ https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1639: --- Assignee: Karl Wright > Upgrade Elastic Search Version > -- > > Key: CONNECTORS-1639 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1639 > Project: ManifoldCF > Issue Type: Improvement > Components: Elastic Search connector >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > Attachments: elastic-search-1.0.1-java11-build-error.log > > > Current Elastic Search version is 1.0.1 . According to [this > matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not > supported by any ES version below 6.5. > Besides, ES 1.x is no longer supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1624) Get ManifoldCF to run under Java 11 or higher
[ https://issues.apache.org/jira/browse/CONNECTORS-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064298#comment-17064298 ] Karl Wright commented on CONNECTORS-1624: - I will request volunteers in dev group for the ElasticSearch task. > Get ManifoldCF to run under Java 11 or higher > - > > Key: CONNECTORS-1624 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1624 > Project: ManifoldCF > Issue Type: Task > Components: Framework core >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > > Java 11 doesn't include a number of classes that Java 8 does. We need to > explicitly include jars that provide these classes or ManifoldCF will not > function under higher Java revs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1637) New Confluence connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053563#comment-17053563
]
Karl Wright commented on CONNECTORS-1637:
-
At the root level:
{code}
ant make-core-deps
ant build
{code}
This will download the dependencies, then build the framework and build all the
connectors. After that you can debug your build locally for your connector by:
{code}
cd connectors/
ant build
{code}
> New Confluence connector
>
>
> Key: CONNECTORS-1637
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1637
> Project: ManifoldCF
> Issue Type: New Feature
> Components: Confluence connector
>Reporter: Julien Massiera
>Assignee: Julien Massiera
>Priority: Major
>
> We need to address 3 main issues of the current Confluence connector :
> - it does not correctly implements the security
> - it has performance problems when handling a huge dataset
> - it generates a version string for documents that is not sufficient to
> detect all changes
> To resolve some of these issues, the connector has to use the new confluence
> API which is available from the v6. For that reason we need to release a new
> connector.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1637) New Confluence connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053389#comment-17053389 ] Karl Wright commented on CONNECTORS-1637: - I saw you committed code in a branch (CONNECTORS-1637) for this. Wonderful! Can you let me know when you think the branch is done? Key things: getting the ant build to work, and making sure ant rat-sources is happy with your Apache headers. > New Confluence connector > > > Key: CONNECTORS-1637 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1637 > Project: ManifoldCF > Issue Type: New Feature > Components: Confluence connector >Reporter: Julien Massiera >Assignee: Julien Massiera >Priority: Major > > We need to address 3 main issues of the current Confluence connector : > - it does not correctly implements the security > - it has performance problems when handling a huge dataset > - it generates a version string for documents that is not sufficient to > detect all changes > To resolve some of these issues, the connector has to use the new confluence > API which is available from the v6. For that reason we need to release a new > connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1629) Support Solr Kerberos Authentication
[ https://issues.apache.org/jira/browse/CONNECTORS-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038017#comment-17038017 ] Karl Wright commented on CONNECTORS-1629: - Great news! I'll close the ticket. > Support Solr Kerberos Authentication > > > Key: CONNECTORS-1629 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1629 > Project: ManifoldCF > Issue Type: Improvement > Components: Lucene/SOLR connector >Affects Versions: ManifoldCF 2.14 >Reporter: Jörn Franke >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > > Several enterprise deployments of Solr are leveraging SolrCloud Kerberos > authentication. > The integration seems to be rather simple and the goal of this Jira is to > evaluate the possential needed step to eventually contribute the Kerberos > integration to the ManifoldCF project. > The following steps would be needed: > * One can pass the JVM parameter java.security.auth.login.config to the > ManifoldCF JVM using -Djava.security.auth.login.config=/path/to/jaas.confg in > which Kerberos authentication details, such as keytab and principal that has > the right access to Solr is configured > * A small adaption to the SolrCloudClient that is used within Manifold needs > to be done to enable Kerberos authentication: > HttpClientUtil.setConfigurer(new Krb5HttpClientConfigurer()); > Should this be integrated in Manifold, one may want to consider one input > field in the configuration in the UI where one can select / flow which user > defined in the Jaas conf (you can define multiple one) should be chosen. By > default one may simply select "client" or "SolrJClient" if Jaas.conf is > present in the System properties. This does not mean the user needs to be > named like this, but the configuration entry referencing any user should be > named like this. > Having a confiugration allows to have a different users per flow. This might > also be needed in case you have multiple Solr clusters. > Related discussion > [http://mail-archives.apache.org/mod_mbox/manifoldcf-user/201912.mbox/browser] > SolrJ Kerberos integration: > [https://lucene.apache.org/solr/guide/8_3/kerberos-authentication-plugin.html#using-solrj-with-a-kerberized-solr] > Jaas conf documentation: > [https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1629) Support Solr Kerberos Authentication
[ https://issues.apache.org/jira/browse/CONNECTORS-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037904#comment-17037904 ] Karl Wright commented on CONNECTORS-1629: - Hi, Your symptom sounds like stuck locks, which can happen if you're using file-based sync and a multiprocess model and you kill processes with kill -9 rather than shutting them down gracefully. > Support Solr Kerberos Authentication > > > Key: CONNECTORS-1629 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1629 > Project: ManifoldCF > Issue Type: Improvement > Components: Lucene/SOLR connector >Affects Versions: ManifoldCF 2.14 >Reporter: Jörn Franke >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > > Several enterprise deployments of Solr are leveraging SolrCloud Kerberos > authentication. > The integration seems to be rather simple and the goal of this Jira is to > evaluate the possential needed step to eventually contribute the Kerberos > integration to the ManifoldCF project. > The following steps would be needed: > * One can pass the JVM parameter java.security.auth.login.config to the > ManifoldCF JVM using -Djava.security.auth.login.config=/path/to/jaas.confg in > which Kerberos authentication details, such as keytab and principal that has > the right access to Solr is configured > * A small adaption to the SolrCloudClient that is used within Manifold needs > to be done to enable Kerberos authentication: > HttpClientUtil.setConfigurer(new Krb5HttpClientConfigurer()); > Should this be integrated in Manifold, one may want to consider one input > field in the configuration in the UI where one can select / flow which user > defined in the Jaas conf (you can define multiple one) should be chosen. By > default one may simply select "client" or "SolrJClient" if Jaas.conf is > present in the System properties. This does not mean the user needs to be > named like this, but the configuration entry referencing any user should be > named like this. > Having a confiugration allows to have a different users per flow. This might > also be needed in case you have multiple Solr clusters. > Related discussion > [http://mail-archives.apache.org/mod_mbox/manifoldcf-user/201912.mbox/browser] > SolrJ Kerberos integration: > [https://lucene.apache.org/solr/guide/8_3/kerberos-authentication-plugin.html#using-solrj-with-a-kerberized-solr] > Jaas conf documentation: > [https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CONNECTORS-1636) ElasticSearch Connector not working with ingest pipeline processor attachment
[
https://issues.apache.org/jira/browse/CONNECTORS-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1636:
---
Assignee: Karl Wright
> ElasticSearch Connector not working with ingest pipeline processor attachment
> -
>
> Key: CONNECTORS-1636
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1636
> Project: ManifoldCF
> Issue Type: Bug
> Components: Elastic Search connector
>Affects Versions: ManifoldCF 2.15
>Reporter: Rohit Batta
>Assignee: Karl Wright
>Priority: Major
> Labels: ManifoldCF, connector, elasticsearch, manifoldcf
> Fix For: ManifoldCF next
>
>
> While using Apache manifoldcf elasticsearch connector for elasticsearch
> version 6.6.x, I found that connector is not working as expected for pipeline
> processor "attachment".
> The processor requires Base64 String to process input stream to content.
> It is working for "mapper-attachment" plugin but that plugin is deprecated in
> newer versions of elasticsearch.
> In case elasticsearch pipeline is used and mapper-attachment is set to false.
> then the content is processed as byte Array to index document, which is not
> correct type for indexing to elasticsearch.
>
> {code:java}
> if (!useMapperAttachments && inputStream != null) {
> if (contentAttributeName != null) {
> Reader r = new InputStreamReader(inputStream, Consts.UTF_8);
> if (needComma) {
> pw.print(",");
> }
> pw.append(jsonStringEscape(contentAttributeName)).append(" : \"");
> char[] buffer = new char[65536];
> while (true) {
> int amt = r.read(buffer, 0, buffer.length);
> if (amt == -1)
> break;
> for (int j = 0; j < amt; j++) {
> final char x = buffer[j];
> if (x == '\n')
> pw.append('\\').append('n');
> else if (x == '\r')
> pw.append('\\').append('r');
> else if (x == '\t')
> pw.append('\\').append('t');
> else if (x == '\b')
> pw.append('\\').append('b');
> else if (x == '\f')
> pw.append('\\').append('f');
> else if (x < 32) {
> pw.append("\\u").append(String.format(Locale.ROOT,
> "%04x", (int) x));
> } else {
> if (x == '\"' || x == '\\' || x == '/')
> pw.append('\\');
> pw.append(x);
> }
> }
> }
> pw.append("\"");
> needComma = true;
> }
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1636) ElasticSearch Connector not working with ingest pipeline processor attachment
[
https://issues.apache.org/jira/browse/CONNECTORS-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037006#comment-17037006
]
Karl Wright commented on CONNECTORS-1636:
-
Patches welcome.
ElasticSearch changes so quickly it's very hard to keep up.
> ElasticSearch Connector not working with ingest pipeline processor attachment
> -
>
> Key: CONNECTORS-1636
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1636
> Project: ManifoldCF
> Issue Type: Bug
> Components: Elastic Search connector
>Affects Versions: ManifoldCF 2.15
>Reporter: Rohit Batta
>Assignee: Karl Wright
>Priority: Major
> Labels: ManifoldCF, connector, elasticsearch, manifoldcf
> Fix For: ManifoldCF next
>
>
> While using Apache manifoldcf elasticsearch connector for elasticsearch
> version 6.6.x, I found that connector is not working as expected for pipeline
> processor "attachment".
> The processor requires Base64 String to process input stream to content.
> It is working for "mapper-attachment" plugin but that plugin is deprecated in
> newer versions of elasticsearch.
> In case elasticsearch pipeline is used and mapper-attachment is set to false.
> then the content is processed as byte Array to index document, which is not
> correct type for indexing to elasticsearch.
>
> {code:java}
> if (!useMapperAttachments && inputStream != null) {
> if (contentAttributeName != null) {
> Reader r = new InputStreamReader(inputStream, Consts.UTF_8);
> if (needComma) {
> pw.print(",");
> }
> pw.append(jsonStringEscape(contentAttributeName)).append(" : \"");
> char[] buffer = new char[65536];
> while (true) {
> int amt = r.read(buffer, 0, buffer.length);
> if (amt == -1)
> break;
> for (int j = 0; j < amt; j++) {
> final char x = buffer[j];
> if (x == '\n')
> pw.append('\\').append('n');
> else if (x == '\r')
> pw.append('\\').append('r');
> else if (x == '\t')
> pw.append('\\').append('t');
> else if (x == '\b')
> pw.append('\\').append('b');
> else if (x == '\f')
> pw.append('\\').append('f');
> else if (x < 32) {
> pw.append("\\u").append(String.format(Locale.ROOT,
> "%04x", (int) x));
> } else {
> if (x == '\"' || x == '\\' || x == '/')
> pw.append('\\');
> pw.append(x);
> }
> }
> }
> pw.append("\"");
> needComma = true;
> }
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1617) Date format extraction problem in XLS/XLSX
[ https://issues.apache.org/jira/browse/CONNECTORS-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036165#comment-17036165 ] Karl Wright commented on CONNECTORS-1617: - Jira -> Create -> pull down "TIKA" in the "Project" pulldown. > Date format extraction problem in XLS/XLSX > -- > > Key: CONNECTORS-1617 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1617 > Project: ManifoldCF > Issue Type: Task > Components: Tika extractor, Tika service connector >Affects Versions: ManifoldCF 2.10 >Reporter: Zoltan Farago >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > Attachments: exceldatum.xlsx > > > Currently TIKA/ManifoldCF 2.10 extracts dates from the attached file tis way: > 2018.05.10 -> 10/05/18 > 2002.02.02 -> 2/2/2 > We need this format: > 2018.05.10 -> 2018-05-10 > 2002.02.02 -> 2002-02-02 > This occurs only when the field type is date. When the field type is text > then the output is fine. > > Please help us with a recommendation with any settings in the pipeline (Tika > configs, excel setting, OS local settings, etc.), or provide a fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CONNECTORS-1617) Date format extraction problem in XLS/XLSX
[ https://issues.apache.org/jira/browse/CONNECTORS-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1617. - Fix Version/s: ManifoldCF 2.16 Resolution: Won't Fix I'm marking this as "won't fix" although it should really be "can't fix". If a Tika ticket gets created to address date format configurability then please include it here; if there's already some configurability present we can work with that. Thanks! > Date format extraction problem in XLS/XLSX > -- > > Key: CONNECTORS-1617 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1617 > Project: ManifoldCF > Issue Type: Task > Components: Tika extractor, Tika service connector >Affects Versions: ManifoldCF 2.10 >Reporter: Zoltan Farago >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > Attachments: exceldatum.xlsx > > > Currently TIKA/ManifoldCF 2.10 extracts dates from the attached file tis way: > 2018.05.10 -> 10/05/18 > 2002.02.02 -> 2/2/2 > We need this format: > 2018.05.10 -> 2018-05-10 > 2002.02.02 -> 2002-02-02 > This occurs only when the field type is date. When the field type is text > then the output is fine. > > Please help us with a recommendation with any settings in the pipeline (Tika > configs, excel setting, OS local settings, etc.), or provide a fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1617) Date format extraction problem in XLS/XLSX
[
https://issues.apache.org/jira/browse/CONNECTORS-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031505#comment-17031505
]
Karl Wright commented on CONNECTORS-1617:
-
The internal Tika extractor treats all metadata as strings, using the Tika
library. I don't think the date format is configurable. Indeed, there's a
blog post on this:
https://grokbase.com/t/tika/user/10982he7yd/how-can-i-configure-tika-to-extract-dates-in-single-format
Note that Tika tries to maintain the date format present in the original
spreadsheet!!
The solution proposed when you want a specific date format is this:
{quote}
* Write your own excel parser for Tika, which ignores the date formatting
set for cells, and always uses iso8601
{quote}
That's not going to cut it here because we don't have any information that
would allow us to autodetect the incoming format properly. It's basically just
a text file and there are no hints, especially for dates like "01-01-2010".
Which comes first, the day or the month?
The external Tika extractor has even less configurability because you cannot
run custom code there.
Now, suppose all you want to do is post-process just *dates* to change the
separator character. Well, we do not know whether the field being returned
from Tika is a date even. If we replaced all /'s with -'s in it then we'd
corrupt other kinds of fields.
My conclusion: there's nothing we can do in ManifoldCF to fix this problem. A
solution might be found in Tika itself, but only if somebody tickets it. Tika
would need to go through the column definitions and understand which columns
were dates and act accordingly. Feel free to open a Tika ticket accordingly.
> Date format extraction problem in XLS/XLSX
> --
>
> Key: CONNECTORS-1617
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1617
> Project: ManifoldCF
> Issue Type: Task
> Components: Tika extractor, Tika service connector
>Affects Versions: ManifoldCF 2.10
>Reporter: Zoltan Farago
>Assignee: Karl Wright
>Priority: Major
> Attachments: exceldatum.xlsx
>
>
> Currently TIKA/ManifoldCF 2.10 extracts dates from the attached file tis way:
> 2018.05.10 -> 10/05/18
> 2002.02.02 -> 2/2/2
> We need this format:
> 2018.05.10 -> 2018-05-10
> 2002.02.02 -> 2002-02-02
> This occurs only when the field type is date. When the field type is text
> then the output is fine.
>
> Please help us with a recommendation with any settings in the pipeline (Tika
> configs, excel setting, OS local settings, etc.), or provide a fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CONNECTORS-1617) Date format extraction problem in XLS/XLSX
[ https://issues.apache.org/jira/browse/CONNECTORS-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1617: --- Assignee: Karl Wright > Date format extraction problem in XLS/XLSX > -- > > Key: CONNECTORS-1617 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1617 > Project: ManifoldCF > Issue Type: Task > Components: Tika extractor, Tika service connector >Affects Versions: ManifoldCF 2.10 >Reporter: Zoltan Farago >Assignee: Karl Wright >Priority: Major > Attachments: exceldatum.xlsx > > > Currently TIKA/ManifoldCF 2.10 extracts dates from the attached file tis way: > 2018.05.10 -> 10/05/18 > 2002.02.02 -> 2/2/2 > We need this format: > 2018.05.10 -> 2018-05-10 > 2002.02.02 -> 2002-02-02 > This occurs only when the field type is date. When the field type is text > then the output is fine. > > Please help us with a recommendation with any settings in the pipeline (Tika > configs, excel setting, OS local settings, etc.), or provide a fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1617) Date format extraction problem in XLS/XLSX
[ https://issues.apache.org/jira/browse/CONNECTORS-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031498#comment-17031498 ] Karl Wright commented on CONNECTORS-1617: - Are you using the external Tika extractor, or the embedded one? > Date format extraction problem in XLS/XLSX > -- > > Key: CONNECTORS-1617 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1617 > Project: ManifoldCF > Issue Type: Task > Components: Tika extractor, Tika service connector >Affects Versions: ManifoldCF 2.10 >Reporter: Zoltan Farago >Assignee: Karl Wright >Priority: Major > Attachments: exceldatum.xlsx > > > Currently TIKA/ManifoldCF 2.10 extracts dates from the attached file tis way: > 2018.05.10 -> 10/05/18 > 2002.02.02 -> 2/2/2 > We need this format: > 2018.05.10 -> 2018-05-10 > 2002.02.02 -> 2002-02-02 > This occurs only when the field type is date. When the field type is text > then the output is fine. > > Please help us with a recommendation with any settings in the pipeline (Tika > configs, excel setting, OS local settings, etc.), or provide a fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CONNECTORS-1597) reflected cross-site scripting vulnerability
[
https://issues.apache.org/jira/browse/CONNECTORS-1597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1597.
-
Fix Version/s: ManifoldCF 2.13
Resolution: Fixed
> reflected cross-site scripting vulnerability
>
>
> Key: CONNECTORS-1597
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1597
> Project: ManifoldCF
> Issue Type: Improvement
> Components: API
>Affects Versions: ManifoldCF 2.12
>Reporter: roel goovaerts
>Assignee: Kishore Kumar
>Priority: Minor
> Fix For: ManifoldCF 2.13
>
>
> This is the full report of a penetration test, performed at a client where we
> deployed a system which uses manifold:
> *Summary*
> A reflected cross-site scripting vulnerability was discovered in the
> application.
> Reflected cross-site scripting occurs when a web application displays data
> submitted by the user that
> contains HTML markup and scripting code without properly escaping it. An
> attacker will create a link to the
> vulnerable page that will display JavaScript code crated by the attacker. The
> attacker will then trick an
> authenticated application user into clicking or following this crated link.
> When the user's browser parses the
> generated page, it will execute the code crafted by the attacker. If the user
> was logged in to the application
> when he followed the link, the attacker's code could perform any action in
> the application that the user can
> perform.
> *Impact*
> Reflected cross-site scripting can be used by attackers to compromise the
> session of an authenticated user.
> By persuading the victim to click on a specially crafted link, the attacker
> can execute his own JavaScript
> payload in the browser context of the victim. In this specific case, an
> attacker could hijack its victim's session
> given that the session token is not flagged as HttpOnly as demonstrated in
> [G190204T1F4][MANIFOLD]
> Insecure Cookie Configuration.
> Additional attacks exist where an attacker can deceive end users of the
> application by redirecting them to
> replica sites or trick them into downloading trojans or other malware. The
> attacker can also use a so called
> browser exploitation framework. In this scenario the attacker injects
> JavaScript code that communicates to
> the attack framework running on the attacker's computer. When the victim user
> executes the JavaScript code
> the attacker can control the victim's browser. Publicly available frameworks
> exist (BeEF -
> [http://www.bindshell.net/tools/beef], Backframe
> -[http://www.gnucitizen.org/projects/backframe/], XSS Proxy -
> [http://xss-proxy.sourceforge.net/]).
> *Affected Systems*
> * [https://els-manifold-uat.bc:8475/mcf-crawler-ui/] [name of an arbitrarily
> supplied URL parameter]
> *Description*
> A case where the application includes user input into the generated HTML
> pages without properly escaping
> the user supplied data was discovered in the application. The HTTP requests
> and responses shown below
> demonstrate the problem.
> {code:java}
> GET /mcf-crawler-ui/?smafi">alert(1)non7x=1 HTTP/1.1
> Host: els-manifold-uat.bc:8475
> Accept-Encoding: gzip, deflate
> Accept: */*
> Accept-Language: en
> User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64;
> Trident/5.0)
> Connection: close
> Cookie: JSESSIONID=ov3qae9biucxdat0xiin5s18
> {code}
> {code:java}
> HTTP/1.1 200 OK
> Server: nginx/1.12.2
> Date: Mon, 18 Feb 2019 13:07:02 GMT
> Content-Type: text/html;charset=utf-8
> Content-Length: 2576
> Connection: close
> Pragma: No-cache
> Expires: Thu, 01 Jan 1970 00:00:00 GMT
> Cache-Control: no-cache
> max-age: Thu, 01 Jan 1970 00:00:00 GMT
>
>
>
> http://www.w3.org/1999/xhtml;>
>
>
>
>
> type="text/css"/>
>
> Apache ManifoldCF⢠Login
>
>
>
>
>
>
>
>
>
>
>
> Sign in to start your session
> method="POST">
> alert(1)non7x=1">
>
> --snip--
> {code}
> *Recommendations*
> We recommend that the application enforces proper validation on user input.
> In most situations where usercontrollable
> data is copied into application responses, cross-site scripting attacks can
> be prevented using two
> layers of defenses:
> * Input should be validated as strictly as possible on arrival, given the
> kind of content which it is
> expected to contain. For example, personal names should consist of
> alphabetical and a small range
> of typographical characters, and be relatively short; a year of birth should
> consist of exactly
[jira] [Commented] (CONNECTORS-1597) reflected cross-site scripting vulnerability
[
https://issues.apache.org/jira/browse/CONNECTORS-1597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028833#comment-17028833
]
Karl Wright commented on CONNECTORS-1597:
-
Final analysis of this ticket is that, since ManifoldCF's UI does not have
different classes of users, and since we've already dealt with any problems
with the login page, escalation of privileges is not a valid attack vector
against the ManifoldCF UI.
> reflected cross-site scripting vulnerability
>
>
> Key: CONNECTORS-1597
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1597
> Project: ManifoldCF
> Issue Type: Improvement
> Components: API
>Affects Versions: ManifoldCF 2.12
>Reporter: roel goovaerts
>Assignee: Kishore Kumar
>Priority: Minor
>
> This is the full report of a penetration test, performed at a client where we
> deployed a system which uses manifold:
> *Summary*
> A reflected cross-site scripting vulnerability was discovered in the
> application.
> Reflected cross-site scripting occurs when a web application displays data
> submitted by the user that
> contains HTML markup and scripting code without properly escaping it. An
> attacker will create a link to the
> vulnerable page that will display JavaScript code crated by the attacker. The
> attacker will then trick an
> authenticated application user into clicking or following this crated link.
> When the user's browser parses the
> generated page, it will execute the code crafted by the attacker. If the user
> was logged in to the application
> when he followed the link, the attacker's code could perform any action in
> the application that the user can
> perform.
> *Impact*
> Reflected cross-site scripting can be used by attackers to compromise the
> session of an authenticated user.
> By persuading the victim to click on a specially crafted link, the attacker
> can execute his own JavaScript
> payload in the browser context of the victim. In this specific case, an
> attacker could hijack its victim's session
> given that the session token is not flagged as HttpOnly as demonstrated in
> [G190204T1F4][MANIFOLD]
> Insecure Cookie Configuration.
> Additional attacks exist where an attacker can deceive end users of the
> application by redirecting them to
> replica sites or trick them into downloading trojans or other malware. The
> attacker can also use a so called
> browser exploitation framework. In this scenario the attacker injects
> JavaScript code that communicates to
> the attack framework running on the attacker's computer. When the victim user
> executes the JavaScript code
> the attacker can control the victim's browser. Publicly available frameworks
> exist (BeEF -
> [http://www.bindshell.net/tools/beef], Backframe
> -[http://www.gnucitizen.org/projects/backframe/], XSS Proxy -
> [http://xss-proxy.sourceforge.net/]).
> *Affected Systems*
> * [https://els-manifold-uat.bc:8475/mcf-crawler-ui/] [name of an arbitrarily
> supplied URL parameter]
> *Description*
> A case where the application includes user input into the generated HTML
> pages without properly escaping
> the user supplied data was discovered in the application. The HTTP requests
> and responses shown below
> demonstrate the problem.
> {code:java}
> GET /mcf-crawler-ui/?smafi">alert(1)non7x=1 HTTP/1.1
> Host: els-manifold-uat.bc:8475
> Accept-Encoding: gzip, deflate
> Accept: */*
> Accept-Language: en
> User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64;
> Trident/5.0)
> Connection: close
> Cookie: JSESSIONID=ov3qae9biucxdat0xiin5s18
> {code}
> {code:java}
> HTTP/1.1 200 OK
> Server: nginx/1.12.2
> Date: Mon, 18 Feb 2019 13:07:02 GMT
> Content-Type: text/html;charset=utf-8
> Content-Length: 2576
> Connection: close
> Pragma: No-cache
> Expires: Thu, 01 Jan 1970 00:00:00 GMT
> Cache-Control: no-cache
> max-age: Thu, 01 Jan 1970 00:00:00 GMT
>
>
>
> http://www.w3.org/1999/xhtml;>
>
>
>
>
> type="text/css"/>
>
> Apache ManifoldCF⢠Login
>
>
>
>
>
>
>
>
>
>
>
> Sign in to start your session
> method="POST">
> alert(1)non7x=1">
>
> --snip--
> {code}
> *Recommendations*
> We recommend that the application enforces proper validation on user input.
> In most situations where usercontrollable
> data is copied into application responses, cross-site scripting attacks can
> be prevented using two
> layers of defenses:
> * Input should be validated as strictly as possible on arrival, given the
> kind of content which it is
> expected
[jira] [Resolved] (CONNECTORS-1595) cross-site request forgery vulnerability
[
https://issues.apache.org/jira/browse/CONNECTORS-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1595.
-
Resolution: Not A Problem
> cross-site request forgery vulnerability
>
>
> Key: CONNECTORS-1595
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1595
> Project: ManifoldCF
> Issue Type: Improvement
> Components: API
>Affects Versions: ManifoldCF 2.12
>Reporter: roel goovaerts
>Assignee: Kishore Kumar
>Priority: Minor
>
> Below is the full analysis and description as a result from the penetration
> test.
> *Summary*
> The application is vulnerable to Cross-Site Request Forgery (CSRF).
> A cross-site request forgery attack uses the following scenario:
> 1. An attacker creates a web page that includes an image or a form pointing
> to the attacked application.
> The image source would actually be a URL with parameters pointing to the
> application page that
> performs some action. In case of a form, the form action would point to the
> action page in the target
> application, and the form is submitted automatically by JavaScript when the
> page is viewed.
> 2. The attacker tricks the victim user to browse to this page. The attacker
> may get the victim to click a
> link, or embed the attacking HTML code into some page the victim views, for
> example in a bulletin
> board or chat.
> 3. When the victim views the attacker's page, his browser sends a request
> prepared by the attacker to
> the attacked application. If the victim is logged in to the target
> application, his browser will possess
> all necessary session tokens, so the request will appear as authorized to the
> application and
> succeed.
> A cross-site request forgery attack uses the fact that the victim's browser
> possesses the necessary
> authentication tokens to perform some actions in the target application.
> *Impact*
> A remote, unauthenticated attacker that can trick an authenticated user into
> clicking a link crafted by the
> attacker or open a malicious web page, can force the victim to unknowingly
> perform various actions within
> the application.
> Given that the whole application is not protected against CSRF, any action
> that an administrator can take on
> Apache Manifold could be unknowingly performed if they fall for a CSRF attack.
> *Affected Systems*
> * [https://els-manifold-uat.bc:8475/mcf-crawler-ui/]
> *Description*
> It appears that the application does not implement any CSRF protection.
> Consider the following example. An
> attacker tricks a logged in application user to visit a page containing the
> following code:
> {code:java}
>
>
>
> history.pushState('', '', '/')
> https://els-manifold-uat.bc:8475/mcf-crawler-ui/execute.jsp;
> method="POST" enctype="multipart/form-data">
>
>
>
>
>
>
> value="orgapachemanifoldcfcrawlerconnectorswebcrawlerWebcr
> awlerConnector" />
>
>
>
> value="ferdiklompcraftworkznl" />
>
>
>
>
>
>
>
>
>
> value="httpsintrauatwebbc" />
> />
>
> value="validation" />
> value=""
> />
> value="Continue" />
> value="username" />
> value="id996812" />
> value="" />
> value="Continue" />
> value="password" />
> value="Th1sIs4cl1X" />
> value="" />
> value="Continue" />
> value="loginformtype" />
> value="pwd" />
> value="" />
> value="3" />
>
>
> value="httpsintrauatwebbc" />
>
>
>
>
>
>
>
>
>
>
>
>
>
> {code}
> When the victim's browser parses the page and tries to load images, it will
> cause them to execute any action
> of the attacker's choosing on Manifold.
> *Recommendations*
> The usual approach to preventing CSRF attacks is to add a new parameter with
> an unpredictable value to
> each form or link that performs some action in the application, commonly
> referred to as a CSRF-Token. The
> parameter value should have enough entropy so that it cannot be predicted by
> an attacker and should be
> unique to the current user session. When the user submits the form or clicks
> the link, the server side code
> checks the parameter value. If it is valid, the request is accepted,
> otherwise it is denied. The attacker has no
> way of knowing the value of the unpredictable parameter, so he cannot
> construct a form or link that will
> submit a valid request.
> *References*
> * OWASP - Cross-Site Request Forgery -
> [https://www.owasp.org/index.php/Cross-]
> Site_Request_Forgery
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1595) cross-site request forgery vulnerability
[
https://issues.apache.org/jira/browse/CONNECTORS-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028832#comment-17028832
]
Karl Wright commented on CONNECTORS-1595:
-
Note: the team here did analyze this potential attack vector, noting the
following:
(1) The UI does not have different capabilities for individual users. The only
security is whether you are "in" or not.
(2) The attack described in this ticket does not get an attacker "in", it just
gives them capabilities once they do that that they might not have. But since
all users are created equally in the MCF UI, this is not problematic.
> cross-site request forgery vulnerability
>
>
> Key: CONNECTORS-1595
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1595
> Project: ManifoldCF
> Issue Type: Improvement
> Components: API
>Affects Versions: ManifoldCF 2.12
>Reporter: roel goovaerts
>Assignee: Kishore Kumar
>Priority: Minor
>
> Below is the full analysis and description as a result from the penetration
> test.
> *Summary*
> The application is vulnerable to Cross-Site Request Forgery (CSRF).
> A cross-site request forgery attack uses the following scenario:
> 1. An attacker creates a web page that includes an image or a form pointing
> to the attacked application.
> The image source would actually be a URL with parameters pointing to the
> application page that
> performs some action. In case of a form, the form action would point to the
> action page in the target
> application, and the form is submitted automatically by JavaScript when the
> page is viewed.
> 2. The attacker tricks the victim user to browse to this page. The attacker
> may get the victim to click a
> link, or embed the attacking HTML code into some page the victim views, for
> example in a bulletin
> board or chat.
> 3. When the victim views the attacker's page, his browser sends a request
> prepared by the attacker to
> the attacked application. If the victim is logged in to the target
> application, his browser will possess
> all necessary session tokens, so the request will appear as authorized to the
> application and
> succeed.
> A cross-site request forgery attack uses the fact that the victim's browser
> possesses the necessary
> authentication tokens to perform some actions in the target application.
> *Impact*
> A remote, unauthenticated attacker that can trick an authenticated user into
> clicking a link crafted by the
> attacker or open a malicious web page, can force the victim to unknowingly
> perform various actions within
> the application.
> Given that the whole application is not protected against CSRF, any action
> that an administrator can take on
> Apache Manifold could be unknowingly performed if they fall for a CSRF attack.
> *Affected Systems*
> * [https://els-manifold-uat.bc:8475/mcf-crawler-ui/]
> *Description*
> It appears that the application does not implement any CSRF protection.
> Consider the following example. An
> attacker tricks a logged in application user to visit a page containing the
> following code:
> {code:java}
>
>
>
> history.pushState('', '', '/')
> https://els-manifold-uat.bc:8475/mcf-crawler-ui/execute.jsp;
> method="POST" enctype="multipart/form-data">
>
>
>
>
>
>
> value="orgapachemanifoldcfcrawlerconnectorswebcrawlerWebcr
> awlerConnector" />
>
>
>
> value="ferdiklompcraftworkznl" />
>
>
>
>
>
>
>
>
>
> value="httpsintrauatwebbc" />
> />
>
> value="validation" />
> value=""
> />
> value="Continue" />
> value="username" />
> value="id996812" />
> value="" />
> value="Continue" />
> value="password" />
> value="Th1sIs4cl1X" />
> value="" />
> value="Continue" />
> value="loginformtype" />
> value="pwd" />
> value="" />
> value="3" />
>
>
> value="httpsintrauatwebbc" />
>
>
>
>
>
>
>
>
>
>
>
>
>
> {code}
> When the victim's browser parses the page and tries to load images, it will
> cause them to execute any action
> of the attacker's choosing on Manifold.
> *Recommendations*
> The usual approach to preventing CSRF attacks is to add a new parameter with
> an unpredictable value to
> each form or link that performs some action in the application, commonly
> referred to as a CSRF-Token. The
> parameter value should have enough entropy so that it cannot be predicted by
> an attacker and should be
> unique to the current user session. When the user submits the form or clicks
> the link, the server side code
> checks the parameter value. If it is valid, the request is accepted,
> otherwise it is denied. The attacker has no
> way of knowing the value of the unpredictable parameter, so he cannot
> construct a form or link that will
> submit a valid request.
> *References*
> * OWASP - Cross-Site Request Forgery -
>
[jira] [Commented] (CONNECTORS-1635) CSWS Connector: Issues with connecting to OpenText system
[ https://issues.apache.org/jira/browse/CONNECTORS-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027255#comment-17027255 ] Karl Wright commented on CONNECTORS-1635: - You're creating a brand-new ResourceResolver() in this fix. Very probably you need to extend the one that is being used rather than creating something brand new, because I bet other methods in the resolver are going to help it resolve links in the wsdl. In other words, you fixed one problem but the way you fixed it broke the ability to fetch the other referenced wsdl components. > CSWS Connector: Issues with connecting to OpenText system > - > > Key: CONNECTORS-1635 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1635 > Project: ManifoldCF > Issue Type: Bug >Affects Versions: ManifoldCF 2.15 >Reporter: Jörn Franke >Assignee: Karl Wright >Priority: Major > > This is about the CSWS connector. It has the following issues: > * Cannot fetch WSDL from https URL, because CA are ignored. The reason is > that the underlying CXF framework uses for fetching the WSDL the library > WSDL4JAVA, which is a completely different mechanism compared to doing a web > service call within the CXF (the latter is correctly addressed by the > connector). See below on how to fix this. > * After fixing fetching WSD from a https URL, another issue occurs. It can > fetch correctly the WSDL, but included references not. The thing is that in > the error message a URL of the included reference is mentioned and this URL > is reachable and also the same server as the WSDL. So I have the theory that > something blocks the CXF request to fetch included files from a https URL. > > Error trace for the second point: > Caused by: org.apache.cxf.service.factory.ServiceConstructionException: > Failed to create service. > at > org.apache.cxf.wsdl11.WSDLServiceFactory.create(WSDLServiceFactory.java:169) > at > org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL(ReflectionServiceFactoryBean.java:408) > at > org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.initializeServiceModel(ReflectionServiceFactoryBean.java:528) > at > org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.create(ReflectionServiceFactoryBean.java:263) > at > org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.create(JaxWsServiceFactoryBean.java:199) > at > org.apache.cxf.frontend.AbstractWSDLBasedEndpointFactory.createEndpoint(AbstractWSDLBasedEndpointFactory.java:103) > at > org.apache.cxf.frontend.ClientFactoryBean.create(ClientFactoryBean.java:91) > at > org.apache.cxf.frontend.ClientProxyFactoryBean.create(ClientProxyFactoryBean.java:159) > at > org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create(JaxWsProxyFactoryBean.java:142) > at org.apache.cxf.jaxws.ServiceImpl.createPort(ServiceImpl.java:492) > at org.apache.cxf.jaxws.ServiceImpl.getPort(ServiceImpl.java:358) > ... 51 more > Caused by: org.apache.ws.commons.schema.XmlSchemaException: Unable to locate > imported document at 'https://server:443/cws/services/Authentication?xsd=2', > relative to > '[https://server:443/cws/services/Authentication?wsdl#types1'|https://deref-web-02.de/mail/client/S_ilqmoKMFI/dereferrer/?redirectUrl=https%3A%2F%2Fd-darwin-dev5.escb.eu%3A443%2Fcws%2Fservices%2FAuthentication%3Fwsdl%23types1%27]. > at > org.apache.cxf.catalog.CatalogXmlSchemaURIResolver.resolveEntity(CatalogXmlSchemaURIResolver.java:76) > at > org.apache.ws.commons.schema.SchemaBuilder.resolveXmlSchema(SchemaBuilder.java:684) > at > org.apache.ws.commons.schema.SchemaBuilder.handleImport(SchemaBuilder.java:538) > at > org.apache.ws.commons.schema.SchemaBuilder.handleSchemaElementChild(SchemaBuilder.java:1515) > at > org.apache.ws.commons.schema.SchemaBuilder.handleXmlSchemaElement(SchemaBuilder.java:658) > at > org.apache.ws.commons.schema.XmlSchemaCollection.read(XmlSchemaCollection.java:550) > at > org.apache.cxf.common.xmlschema.SchemaCollection.read(SchemaCollection.java:129) > at org.apache.cxf.wsdl11.SchemaUtil.extractSchema(SchemaUtil.java:141) > at org.apache.cxf.wsdl11.SchemaUtil.getSchemas(SchemaUtil.java:74) > at org.apache.cxf.wsdl11.SchemaUtil.getSchemas(SchemaUtil.java:66) > at org.apache.cxf.wsdl11.SchemaUtil.getSchemas(SchemaUtil.java:61) > at > org.apache.cxf.wsdl11.WSDLServiceBuilder.getSchemas(WSDLServiceBuilder.java:378) > at > org.apache.cxf.wsdl11.WSDLServiceBuilder.buildServices(WSDLServiceBuilder.java:345) > at > org.apache.cxf.wsdl11.WSDLServiceBuilder.buildServices(WSDLServiceBuilder.java:209) > at >
[jira] [Assigned] (CONNECTORS-1635) CSWS Connector: Issues with connecting to OpenText system
[
https://issues.apache.org/jira/browse/CONNECTORS-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1635:
---
Assignee: Karl Wright
> CSWS Connector: Issues with connecting to OpenText system
> -
>
> Key: CONNECTORS-1635
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1635
> Project: ManifoldCF
> Issue Type: Bug
>Affects Versions: ManifoldCF 2.15
>Reporter: Jörn Franke
>Assignee: Karl Wright
>Priority: Major
>
> This is about the CSWS connector. It has the following issues:
> * Cannot fetch WSDL from https URL, because CA are ignored. The reason is
> that the underlying CXF framework uses for fetching the WSDL the library
> WSDL4JAVA, which is a completely different mechanism compared to doing a web
> service call within the CXF (the latter is correctly addressed by the
> connector). See below on how to fix this.
> * After fixing fetching WSD from a https URL, another issue occurs. It can
> fetch correctly the WSDL, but included references not. The thing is that in
> the error message a URL of the included reference is mentioned and this URL
> is reachable and also the same server as the WSDL. So I have the theory that
> something blocks the CXF request to fetch included files from a https URL.
>
> Error trace for the second point:
> Caused by: org.apache.cxf.service.factory.ServiceConstructionException:
> Failed to create service.
> at
> org.apache.cxf.wsdl11.WSDLServiceFactory.create(WSDLServiceFactory.java:169)
> at
> org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL(ReflectionServiceFactoryBean.java:408)
> at
> org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.initializeServiceModel(ReflectionServiceFactoryBean.java:528)
> at
> org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.create(ReflectionServiceFactoryBean.java:263)
> at
> org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.create(JaxWsServiceFactoryBean.java:199)
> at
> org.apache.cxf.frontend.AbstractWSDLBasedEndpointFactory.createEndpoint(AbstractWSDLBasedEndpointFactory.java:103)
> at
> org.apache.cxf.frontend.ClientFactoryBean.create(ClientFactoryBean.java:91)
> at
> org.apache.cxf.frontend.ClientProxyFactoryBean.create(ClientProxyFactoryBean.java:159)
> at
> org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create(JaxWsProxyFactoryBean.java:142)
> at org.apache.cxf.jaxws.ServiceImpl.createPort(ServiceImpl.java:492)
> at org.apache.cxf.jaxws.ServiceImpl.getPort(ServiceImpl.java:358)
> ... 51 more
> Caused by: org.apache.ws.commons.schema.XmlSchemaException: Unable to locate
> imported document at 'https://server:443/cws/services/Authentication?xsd=2',
> relative to
> '[https://server:443/cws/services/Authentication?wsdl#types1'|https://deref-web-02.de/mail/client/S_ilqmoKMFI/dereferrer/?redirectUrl=https%3A%2F%2Fd-darwin-dev5.escb.eu%3A443%2Fcws%2Fservices%2FAuthentication%3Fwsdl%23types1%27].
> at
> org.apache.cxf.catalog.CatalogXmlSchemaURIResolver.resolveEntity(CatalogXmlSchemaURIResolver.java:76)
> at
> org.apache.ws.commons.schema.SchemaBuilder.resolveXmlSchema(SchemaBuilder.java:684)
> at
> org.apache.ws.commons.schema.SchemaBuilder.handleImport(SchemaBuilder.java:538)
> at
> org.apache.ws.commons.schema.SchemaBuilder.handleSchemaElementChild(SchemaBuilder.java:1515)
> at
> org.apache.ws.commons.schema.SchemaBuilder.handleXmlSchemaElement(SchemaBuilder.java:658)
> at
> org.apache.ws.commons.schema.XmlSchemaCollection.read(XmlSchemaCollection.java:550)
> at
> org.apache.cxf.common.xmlschema.SchemaCollection.read(SchemaCollection.java:129)
> at org.apache.cxf.wsdl11.SchemaUtil.extractSchema(SchemaUtil.java:141)
> at org.apache.cxf.wsdl11.SchemaUtil.getSchemas(SchemaUtil.java:74)
> at org.apache.cxf.wsdl11.SchemaUtil.getSchemas(SchemaUtil.java:66)
> at org.apache.cxf.wsdl11.SchemaUtil.getSchemas(SchemaUtil.java:61)
> at
> org.apache.cxf.wsdl11.WSDLServiceBuilder.getSchemas(WSDLServiceBuilder.java:378)
> at
> org.apache.cxf.wsdl11.WSDLServiceBuilder.buildServices(WSDLServiceBuilder.java:345)
> at
> org.apache.cxf.wsdl11.WSDLServiceBuilder.buildServices(WSDLServiceBuilder.java:209)
> at
> org.apache.cxf.wsdl11.WSDLServiceFactory.create(WSDLServiceFactory.java:161)
> ... 61 more
>
>
>
> ---
> fixing https CAs for fetching WSDLs:
> // enable https for wsdl requests (this goes via WSDL4J)
> Bus bus = BusFactory.getThreadDefaultBus();
> ResourceManager extension =
> bus.getExtension(ResourceManager.class);
> extension.addResourceResolver(new ResourceResolver() {
> @Override
>
[jira] [Commented] (CONNECTORS-1622) Upgrade to Tika 1.23
[
https://issues.apache.org/jira/browse/CONNECTORS-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024054#comment-17024054
]
Karl Wright commented on CONNECTORS-1622:
-
In case anyone's curious what the proper process is for this, what I do is
change the pom.xml version for Tika first, then do the following steps:
{code}
mvn -DskipTests -DskipITs install
mvn dependency:tree
{code}
Then I go through the tika dependency tree to update all the packages, add any
that are missing, etc., in the main build.xml file.
> Upgrade to Tika 1.23
>
>
> Key: CONNECTORS-1622
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1622
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Tika extractor
>Affects Versions: ManifoldCF 2.13
>Reporter: Cihad Guzel
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF next
>
>
> Tika has released 1.23. Changes can be found from here:
> http://tika.apache.org/1.23/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CONNECTORS-1634) Occured 501 HTTP Error while downloading 'h2' dependency
[
https://issues.apache.org/jira/browse/CONNECTORS-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1634.
-
Resolution: Fixed
r1873157
> Occured 501 HTTP Error while downloading 'h2' dependency
>
>
> Key: CONNECTORS-1634
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1634
> Project: ManifoldCF
> Issue Type: Improvement
>Affects Versions: ManifoldCF 2.15
>Reporter: Cihad Guzel
>Assignee: Cihad Guzel
>Priority: Major
> Fix For: ManifoldCF 2.16
>
>
> We have to update h2 dependency download link. I try this command:
> {noformat}
> ant make-core-deps{noformat}
> I have this error:
> {noformat}
> download-h2:
> [get] Getting:
> http://repo2.maven.org/maven2/com/h2database/h2/1.3.158/h2-1.3.158.jar
> [get] To: /Users/cguzel/Projects/apache/svn/mcf-trunk/lib/h2-1.3.158.jar
> [get] Error opening connection java.io.IOException: Server returned
> HTTP response code: 501 for URL:
> http://repo2.maven.org/maven2/com/h2database/h2/1.3.158/h2-1.3.158.jar
> [get] Error opening connection java.io.IOException: Server returned
> HTTP response code: 501 for URL:
> http://repo2.maven.org/maven2/com/h2database/h2/1.3.158/h2-1.3.158.jar
> [get] Error opening connection java.io.IOException: Server returned
> HTTP response code: 501 for URL:
> http://repo2.maven.org/maven2/com/h2database/h2/1.3.158/h2-1.3.158.jar
> [get] Can't get
> http://repo2.maven.org/maven2/com/h2database/h2/1.3.158/h2-1.3.158.jar to
> /Users/cguzel/Projects/apache/svn/mcf-trunk/lib/h2-1.3.158.jar
> BUILD FAILED
> /Users/cguzel/Projects/apache/svn/mcf-trunk/build.xml:1514: Can't get
> http://repo2.maven.org/maven2/com/h2database/h2/1.3.158/h2-1.3.158.jar to
> /Users/cguzel/Projects/apache/svn/mcf-trunk/lib/h2-1.3.158.jar
> {noformat}
> I tried the request in my browser and got the following response:
> [http://repo2.maven.org/maven2/com/h2database/h2/1.3.158/h2-1.3.158.jar]
> {noformat}
> 501 HTTPS Required.
> Use https://repo1.maven.org/maven2/
> More information at https://links.sonatype.com/central/501-https-required
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1634) Occured 501 HTTP Error while downloading 'h2' dependency
[
https://issues.apache.org/jira/browse/CONNECTORS-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023606#comment-17023606
]
Karl Wright commented on CONNECTORS-1634:
-
Let me try this here. We released in December and there was no issue then.
> Occured 501 HTTP Error while downloading 'h2' dependency
>
>
> Key: CONNECTORS-1634
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1634
> Project: ManifoldCF
> Issue Type: Improvement
>Affects Versions: ManifoldCF 2.15
>Reporter: Cihad Guzel
>Assignee: Cihad Guzel
>Priority: Major
> Fix For: ManifoldCF 2.16
>
>
> We have to update h2 dependency download link. I try this command:
> {noformat}
> ant make-core-deps{noformat}
> I have this error:
> {noformat}
> download-h2:
> [get] Getting:
> http://repo2.maven.org/maven2/com/h2database/h2/1.3.158/h2-1.3.158.jar
> [get] To: /Users/cguzel/Projects/apache/svn/mcf-trunk/lib/h2-1.3.158.jar
> [get] Error opening connection java.io.IOException: Server returned
> HTTP response code: 501 for URL:
> http://repo2.maven.org/maven2/com/h2database/h2/1.3.158/h2-1.3.158.jar
> [get] Error opening connection java.io.IOException: Server returned
> HTTP response code: 501 for URL:
> http://repo2.maven.org/maven2/com/h2database/h2/1.3.158/h2-1.3.158.jar
> [get] Error opening connection java.io.IOException: Server returned
> HTTP response code: 501 for URL:
> http://repo2.maven.org/maven2/com/h2database/h2/1.3.158/h2-1.3.158.jar
> [get] Can't get
> http://repo2.maven.org/maven2/com/h2database/h2/1.3.158/h2-1.3.158.jar to
> /Users/cguzel/Projects/apache/svn/mcf-trunk/lib/h2-1.3.158.jar
> BUILD FAILED
> /Users/cguzel/Projects/apache/svn/mcf-trunk/build.xml:1514: Can't get
> http://repo2.maven.org/maven2/com/h2database/h2/1.3.158/h2-1.3.158.jar to
> /Users/cguzel/Projects/apache/svn/mcf-trunk/lib/h2-1.3.158.jar
> {noformat}
> I tried the request in my browser and got the following response:
> [http://repo2.maven.org/maven2/com/h2database/h2/1.3.158/h2-1.3.158.jar]
> {noformat}
> 501 HTTPS Required.
> Use https://repo1.maven.org/maven2/
> More information at https://links.sonatype.com/central/501-https-required
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1629) Support Solr Kerberos Authentication
[ https://issues.apache.org/jira/browse/CONNECTORS-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023495#comment-17023495 ] Karl Wright commented on CONNECTORS-1629: - [~jornfranke], I can add the documentation. But what I want you to do is re-test, since I changed some things around. Specifically, does it work to reference the jaas-config file by using a relative path, e.g. "./jaas-client.config"? I believe it should but needs to be confirmed. That's simple to add to the instructions. Also, if you include a quick description in your own words (with online references as needed) for how to edit jaas-client.config to meet your own needs, I can edit it accordingly. Please just include as a comment in this ticket. > Support Solr Kerberos Authentication > > > Key: CONNECTORS-1629 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1629 > Project: ManifoldCF > Issue Type: Improvement > Components: Lucene/SOLR connector >Affects Versions: ManifoldCF 2.14 >Reporter: Jörn Franke >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > > Several enterprise deployments of Solr are leveraging SolrCloud Kerberos > authentication. > The integration seems to be rather simple and the goal of this Jira is to > evaluate the possential needed step to eventually contribute the Kerberos > integration to the ManifoldCF project. > The following steps would be needed: > * One can pass the JVM parameter java.security.auth.login.config to the > ManifoldCF JVM using -Djava.security.auth.login.config=/path/to/jaas.confg in > which Kerberos authentication details, such as keytab and principal that has > the right access to Solr is configured > * A small adaption to the SolrCloudClient that is used within Manifold needs > to be done to enable Kerberos authentication: > HttpClientUtil.setConfigurer(new Krb5HttpClientConfigurer()); > Should this be integrated in Manifold, one may want to consider one input > field in the configuration in the UI where one can select / flow which user > defined in the Jaas conf (you can define multiple one) should be chosen. By > default one may simply select "client" or "SolrJClient" if Jaas.conf is > present in the System properties. This does not mean the user needs to be > named like this, but the configuration entry referencing any user should be > named like this. > Having a confiugration allows to have a different users per flow. This might > also be needed in case you have multiple Solr clusters. > Related discussion > [http://mail-archives.apache.org/mod_mbox/manifoldcf-user/201912.mbox/browser] > SolrJ Kerberos integration: > [https://lucene.apache.org/solr/guide/8_3/kerberos-authentication-plugin.html#using-solrj-with-a-kerberized-solr] > Jaas conf documentation: > [https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1624) Get ManifoldCF to run under Java 11 or higher
[ https://issues.apache.org/jira/browse/CONNECTORS-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023487#comment-17023487 ] Karl Wright commented on CONNECTORS-1624: - [~cguzel], this needs to be worked on this release. I'd like to know what set of packages from the Maven Repository will replace what's removed from JDK 11 vs. JDK 8. If you have time to research that and provide that information in this ticket I'd be very grateful. Thanks in advance! > Get ManifoldCF to run under Java 11 or higher > - > > Key: CONNECTORS-1624 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1624 > Project: ManifoldCF > Issue Type: Task > Components: Framework core >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > > Java 11 doesn't include a number of classes that Java 8 does. We need to > explicitly include jars that provide these classes or ManifoldCF will not > function under higher Java revs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CONNECTORS-1624) Get ManifoldCF to run under Java 11 or higher
[ https://issues.apache.org/jira/browse/CONNECTORS-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright updated CONNECTORS-1624: Fix Version/s: (was: ManifoldCF next) ManifoldCF 2.16 > Get ManifoldCF to run under Java 11 or higher > - > > Key: CONNECTORS-1624 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1624 > Project: ManifoldCF > Issue Type: Task > Components: Framework core >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > > Java 11 doesn't include a number of classes that Java 8 does. We need to > explicitly include jars that provide these classes or ManifoldCF will not > function under higher Java revs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CONNECTORS-1629) Support Solr Kerberos Authentication
[ https://issues.apache.org/jira/browse/CONNECTORS-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright updated CONNECTORS-1629: Component/s: (was: Solr 7.x component) Lucene/SOLR connector > Support Solr Kerberos Authentication > > > Key: CONNECTORS-1629 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1629 > Project: ManifoldCF > Issue Type: Improvement > Components: Lucene/SOLR connector >Affects Versions: ManifoldCF 2.14 >Reporter: Jörn Franke >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > > Several enterprise deployments of Solr are leveraging SolrCloud Kerberos > authentication. > The integration seems to be rather simple and the goal of this Jira is to > evaluate the possential needed step to eventually contribute the Kerberos > integration to the ManifoldCF project. > The following steps would be needed: > * One can pass the JVM parameter java.security.auth.login.config to the > ManifoldCF JVM using -Djava.security.auth.login.config=/path/to/jaas.confg in > which Kerberos authentication details, such as keytab and principal that has > the right access to Solr is configured > * A small adaption to the SolrCloudClient that is used within Manifold needs > to be done to enable Kerberos authentication: > HttpClientUtil.setConfigurer(new Krb5HttpClientConfigurer()); > Should this be integrated in Manifold, one may want to consider one input > field in the configuration in the UI where one can select / flow which user > defined in the Jaas conf (you can define multiple one) should be chosen. By > default one may simply select "client" or "SolrJClient" if Jaas.conf is > present in the System properties. This does not mean the user needs to be > named like this, but the configuration entry referencing any user should be > named like this. > Having a confiugration allows to have a different users per flow. This might > also be needed in case you have multiple Solr clusters. > Related discussion > [http://mail-archives.apache.org/mod_mbox/manifoldcf-user/201912.mbox/browser] > SolrJ Kerberos integration: > [https://lucene.apache.org/solr/guide/8_3/kerberos-authentication-plugin.html#using-solrj-with-a-kerberized-solr] > Jaas conf documentation: > [https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CONNECTORS-1629) Support Solr Kerberos Authentication
[ https://issues.apache.org/jira/browse/CONNECTORS-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1629. - Fix Version/s: ManifoldCF 2.16 Resolution: Fixed Still need documentation improvement on the "how to build and deploy" page > Support Solr Kerberos Authentication > > > Key: CONNECTORS-1629 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1629 > Project: ManifoldCF > Issue Type: Improvement > Components: Solr 7.x component >Affects Versions: ManifoldCF 2.14 >Reporter: Jörn Franke >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > > Several enterprise deployments of Solr are leveraging SolrCloud Kerberos > authentication. > The integration seems to be rather simple and the goal of this Jira is to > evaluate the possential needed step to eventually contribute the Kerberos > integration to the ManifoldCF project. > The following steps would be needed: > * One can pass the JVM parameter java.security.auth.login.config to the > ManifoldCF JVM using -Djava.security.auth.login.config=/path/to/jaas.confg in > which Kerberos authentication details, such as keytab and principal that has > the right access to Solr is configured > * A small adaption to the SolrCloudClient that is used within Manifold needs > to be done to enable Kerberos authentication: > HttpClientUtil.setConfigurer(new Krb5HttpClientConfigurer()); > Should this be integrated in Manifold, one may want to consider one input > field in the configuration in the UI where one can select / flow which user > defined in the Jaas conf (you can define multiple one) should be chosen. By > default one may simply select "client" or "SolrJClient" if Jaas.conf is > present in the System properties. This does not mean the user needs to be > named like this, but the configuration entry referencing any user should be > named like this. > Having a confiugration allows to have a different users per flow. This might > also be needed in case you have multiple Solr clusters. > Related discussion > [http://mail-archives.apache.org/mod_mbox/manifoldcf-user/201912.mbox/browser] > SolrJ Kerberos integration: > [https://lucene.apache.org/solr/guide/8_3/kerberos-authentication-plugin.html#using-solrj-with-a-kerberized-solr] > Jaas conf documentation: > [https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CONNECTORS-1521) Documentum Connector users ManifoldCF's local time in queries constraints against the Documentum server without reference to time zones
[
https://issues.apache.org/jira/browse/CONNECTORS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright updated CONNECTORS-1521:
Fix Version/s: (was: ManifoldCF 2.15)
ManifoldCF next
> Documentum Connector users ManifoldCF's local time in queries constraints
> against the Documentum server without reference to time zones
> ---
>
> Key: CONNECTORS-1521
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1521
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF next
>
>
> I find that the time/date constraints in queries to the Documentum server are
> based on the "raw" local time of the ManifoldCF server but appear to take no
> account of the time zones of the two servers.
> This can lead to recently modified files not being transferred to the output
> repository when you would naturally expect them to be. I'd like the times to
> be aligned, perhaps by including time zone in the query. In particular, is
> there a way to use UTC perhaps?
> Here's an example ...
> * create a folder in Documentum
> * set up a job to point at the folder and output to the file system
> * put two documents into a folder in Documentum
> * Select them, right click and export as CSV (to show the timestamps):
> {noformat}
> 1.png,48489.0,Portable Network Graphics,8/7/2018 9:04 AM,
> 2.png,28620.0,Portable Network Graphics,8/7/2018 9:04 AM,,{noformat}
> Check the local time on the ManifoldCF server machine. Observe that it's
> reporting consistent time with the DM server:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 09:07:25 BST 2018{noformat}
> Start the job and look for the query to Documentum in the manifoldcf.log file
> (line break added for readability):
> {noformat}
> DEBUG 2018-08-07T08:07:47.297Z (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 00:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('08/07/2018 08:07:34','mm/dd/ hh:mi:ss')
> AND (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0)) AND ( Folder('/Administrator/james', DESCEND) ))
> ^C{noformat}
> Notice that the latest date asked for is *before* the modification date of
> the files added to DM. (And is an hour out, see footnote.)
>
> See whether anything has been output by the File System connector. It hasn't:
> {noformat}
> [james@manifold]$ ls /bigdisc/source/PDFs/timezones/
> [james@manifold]$
> {noformat}
> Now:
> * change the timezone on the ManifoldCF server machine
> * restart the ManifoldCF server and the Documentum processes
> * reseed the job
> Check the local time on the ManifoldCF server machine; it has changed:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 10:10:29 CEST 2018{noformat}
> Start the job again and notice that the query has changed by an hour, plus
> the few minutes it took to change the date etc (and is still an hour out, see
> footnote):
> {noformat}
> r_modify_date<=date('08/07/2018 09:11:02','mm/dd/ hh:mi:ss')
> {noformat}
> Observe that the range of dates now covers the timestamps on the DM data, and
> also that some data has now been transferred by the File System connector:
> {noformat}
> [james@manifold]$ ls
> /bigdisc/source/PDFs/timezones/http/mfserver\:8080/da/component/
> drl?versionLabel=CURRENT=09018000e515
> drl?versionLabel=CURRENT=09018000e516
> {noformat}
>
>
> [Footnote] It appears that something is trying to take account of Daylight
> Saving Time too.
> If I set the server date to a time outside of DST, the query is aligned with
> the current time:
> {noformat}
> [i2e@i2ehost manifold]$ date
> Mon Oct 29 00:01:13 CET 2018
> r_modify_date<=date('10/29/2018 00:01:39','mm/dd/ hh:mi:ss')
> {noformat}
> But if I set the time inside DST, the time is an hour before:
> {noformat}
> [i2e@i2ehost manifold]$ date
> Sat Oct 27 00:00:06 CEST 2018
> r_modify_date<=date('10/26/2018 23:00:26','mm/dd/ hh:mi:ss')
> {noformat}
> This is perhaps a Java issue rather than a logic issue in the connector? See
> e.g. [https://stackoverflow.com/questions/6392/java-time-zone-is-messed-up]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CONNECTORS-1624) Get ManifoldCF to run under Java 11 or higher
[ https://issues.apache.org/jira/browse/CONNECTORS-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright updated CONNECTORS-1624: Fix Version/s: (was: ManifoldCF 2.15) ManifoldCF next > Get ManifoldCF to run under Java 11 or higher > - > > Key: CONNECTORS-1624 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1624 > Project: ManifoldCF > Issue Type: Task > Components: Framework core >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF next > > > Java 11 doesn't include a number of classes that Java 8 does. We need to > explicitly include jars that provide these classes or ManifoldCF will not > function under higher Java revs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CONNECTORS-1508) Add support for French Language
[ https://issues.apache.org/jira/browse/CONNECTORS-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright updated CONNECTORS-1508: Fix Version/s: (was: ManifoldCF 2.15) ManifoldCF next > Add support for French Language > --- > > Key: CONNECTORS-1508 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1508 > Project: ManifoldCF > Issue Type: Improvement > Components: Documentation >Affects Versions: ManifoldCF 2.10 >Reporter: Cedric Ulmer >Assignee: Karl Wright >Priority: Minor > Fix For: ManifoldCF next > > Attachments: cedricmanifold_fr.zip > > > Some users may need a French version of the ressource bundle. I attached a > preliminary translation that France Labs made some time ago (probably around > summer 2016), but that we halted due to lack of time (and priority). It is > probably almost complete, but some quality checking needs to be done. Note > also that I forgot to check the version when I did the translations, so > anyone interested would need to check any modifications that may have > occurred between this version and the current MCF version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CONNECTORS-1622) Upgrade to Tika 1.22
[ https://issues.apache.org/jira/browse/CONNECTORS-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright updated CONNECTORS-1622: Fix Version/s: (was: ManifoldCF 2.15) ManifoldCF next > Upgrade to Tika 1.22 > > > Key: CONNECTORS-1622 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1622 > Project: ManifoldCF > Issue Type: Improvement > Components: Tika extractor >Affects Versions: ManifoldCF 2.13 >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF next > > > Tika has released 1.22. Changes can be found from here: > http://tika.apache.org/1.22/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1629) Support Solr Kerberos Authentication
[ https://issues.apache.org/jira/browse/CONNECTORS-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023381#comment-17023381 ] Karl Wright commented on CONNECTORS-1629: - r1873121 commits this code and includes stub -D switches wherever they are needed. I've included the sample jaas-config file but please note that the options.env files need to be hand-modified to point to the jaas-config file to enable Kerberos. There's a placeholder empty -D that should be completed. This deserves mention in the "how-to-build-and-deploy" page, which I will add as soon as we reverify that everything works as expected still when built from trunk. > Support Solr Kerberos Authentication > > > Key: CONNECTORS-1629 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1629 > Project: ManifoldCF > Issue Type: Improvement > Components: Solr 7.x component >Affects Versions: ManifoldCF 2.14 >Reporter: Jörn Franke >Assignee: Karl Wright >Priority: Major > > Several enterprise deployments of Solr are leveraging SolrCloud Kerberos > authentication. > The integration seems to be rather simple and the goal of this Jira is to > evaluate the possential needed step to eventually contribute the Kerberos > integration to the ManifoldCF project. > The following steps would be needed: > * One can pass the JVM parameter java.security.auth.login.config to the > ManifoldCF JVM using -Djava.security.auth.login.config=/path/to/jaas.confg in > which Kerberos authentication details, such as keytab and principal that has > the right access to Solr is configured > * A small adaption to the SolrCloudClient that is used within Manifold needs > to be done to enable Kerberos authentication: > HttpClientUtil.setConfigurer(new Krb5HttpClientConfigurer()); > Should this be integrated in Manifold, one may want to consider one input > field in the configuration in the UI where one can select / flow which user > defined in the Jaas conf (you can define multiple one) should be chosen. By > default one may simply select "client" or "SolrJClient" if Jaas.conf is > present in the System properties. This does not mean the user needs to be > named like this, but the configuration entry referencing any user should be > named like this. > Having a confiugration allows to have a different users per flow. This might > also be needed in case you have multiple Solr clusters. > Related discussion > [http://mail-archives.apache.org/mod_mbox/manifoldcf-user/201912.mbox/browser] > SolrJ Kerberos integration: > [https://lucene.apache.org/solr/guide/8_3/kerberos-authentication-plugin.html#using-solrj-with-a-kerberized-solr] > Jaas conf documentation: > [https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1633) Exception tossed: Repeated service interruptions - failure processing document: The process cannot access the file because it is being used by another process.
[
https://issues.apache.org/jira/browse/CONNECTORS-1633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022959#comment-17022959
]
Karl Wright commented on CONNECTORS-1633:
-
Hi,
The connector retries for a specific period of time on this and then gives up
and aborts the job. What kind of behavior would you like to see different? It
could choose to skip the file and continue instead, but that I'd worry about
too.
> Exception tossed: Repeated service interruptions - failure processing
> document: The process cannot access the file because it is being used by
> another process.
> ---
>
> Key: CONNECTORS-1633
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1633
> Project: ManifoldCF
> Issue Type: Bug
> Components: File system connector
>Affects Versions: ManifoldCF 2.13
>Reporter: Michael Cizmar
>Assignee: Karl Wright
>Priority: Major
>
> Seeing this error occurring and I'm working to address it. If it's not a
> bug, a better message should be generated.
>
> {code:java}
> crawl job fails with the following error due to document being in use by
> another user:
> WARN 2019-08-25T15:02:27,416 (Worker thread '11') - Service interruption
> reported for job 1565115290083 connection 'fs_vwoaahvp319': Timeout or other
> service interruption: The process cannot access the file because it is being
> used by another process.
> ERROR 2019-08-25T15:02:27,424 (Worker thread '11') - Exception tossed:
> Repeated service interruptions - failure processing document: The process
> cannot access the file because it is being used by another process.
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Repeated service
> interruptions - failure processing document: The process cannot access the
> file because it is being used by another process.
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:489)
> [mcf-pull-agent.jar:?]
> Caused by: jcifs.smb.SmbException: The process cannot access the file because
> it is being used by another process.
> at
> jcifs.smb.SmbTransportImpl.checkStatus2(SmbTransportImpl.java:1457) ~[?:?]
> at jcifs.smb.SmbTransportImpl.checkStatus(SmbTransportImpl.java:1568)
> ~[?:?]
> at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1023)
> ~[?:?]
> at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1539) ~[?:?]
> at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409) ~[?:?]
> at jcifs.smb.SmbTreeImpl.send(SmbTreeImpl.java:472) ~[?:?]
> at jcifs.smb.SmbTreeConnection.send0(SmbTreeConnection.java:401)
> ~[?:?]
> at jcifs.smb.SmbTreeConnection.send(SmbTreeConnection.java:315) ~[?:?]
> at jcifs.smb.SmbTreeConnection.send(SmbTreeConnection.java:295) ~[?:?]
> at jcifs.smb.SmbTreeHandleImpl.send(SmbTreeHandleImpl.java:130) ~[?:?]
> at jcifs.smb.SmbTreeHandleImpl.send(SmbTreeHandleImpl.java:117) ~[?:?]
> at jcifs.smb.SmbFile.withOpen(SmbFile.java:1741) ~[?:?]
> at jcifs.smb.SmbFile.withOpen(SmbFile.java:1710) ~[?:?]
> at jcifs.smb.SmbFile.withOpen(SmbFile.java:1704) ~[?:?]
> at jcifs.smb.SmbFile.queryPath(SmbFile.java:770) ~[?:?]
> at jcifs.smb.SmbFile.exists(SmbFile.java:851) ~[?:?]
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2188)
> ~[?:?]
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:610)
> ~[?:?]
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> ~[mcf-pull-agent.jar:?]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CONNECTORS-1633) Exception tossed: Repeated service interruptions - failure processing document: The process cannot access the file because it is being used by another process.
[
https://issues.apache.org/jira/browse/CONNECTORS-1633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1633:
---
Assignee: Karl Wright
> Exception tossed: Repeated service interruptions - failure processing
> document: The process cannot access the file because it is being used by
> another process.
> ---
>
> Key: CONNECTORS-1633
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1633
> Project: ManifoldCF
> Issue Type: Bug
> Components: File system connector
>Affects Versions: ManifoldCF 2.13
>Reporter: Michael Cizmar
>Assignee: Karl Wright
>Priority: Major
>
> Seeing this error occurring and I'm working to address it. If it's not a
> bug, a better message should be generated.
>
> {code:java}
> crawl job fails with the following error due to document being in use by
> another user:
> WARN 2019-08-25T15:02:27,416 (Worker thread '11') - Service interruption
> reported for job 1565115290083 connection 'fs_vwoaahvp319': Timeout or other
> service interruption: The process cannot access the file because it is being
> used by another process.
> ERROR 2019-08-25T15:02:27,424 (Worker thread '11') - Exception tossed:
> Repeated service interruptions - failure processing document: The process
> cannot access the file because it is being used by another process.
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Repeated service
> interruptions - failure processing document: The process cannot access the
> file because it is being used by another process.
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:489)
> [mcf-pull-agent.jar:?]
> Caused by: jcifs.smb.SmbException: The process cannot access the file because
> it is being used by another process.
> at
> jcifs.smb.SmbTransportImpl.checkStatus2(SmbTransportImpl.java:1457) ~[?:?]
> at jcifs.smb.SmbTransportImpl.checkStatus(SmbTransportImpl.java:1568)
> ~[?:?]
> at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1023)
> ~[?:?]
> at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1539) ~[?:?]
> at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409) ~[?:?]
> at jcifs.smb.SmbTreeImpl.send(SmbTreeImpl.java:472) ~[?:?]
> at jcifs.smb.SmbTreeConnection.send0(SmbTreeConnection.java:401)
> ~[?:?]
> at jcifs.smb.SmbTreeConnection.send(SmbTreeConnection.java:315) ~[?:?]
> at jcifs.smb.SmbTreeConnection.send(SmbTreeConnection.java:295) ~[?:?]
> at jcifs.smb.SmbTreeHandleImpl.send(SmbTreeHandleImpl.java:130) ~[?:?]
> at jcifs.smb.SmbTreeHandleImpl.send(SmbTreeHandleImpl.java:117) ~[?:?]
> at jcifs.smb.SmbFile.withOpen(SmbFile.java:1741) ~[?:?]
> at jcifs.smb.SmbFile.withOpen(SmbFile.java:1710) ~[?:?]
> at jcifs.smb.SmbFile.withOpen(SmbFile.java:1704) ~[?:?]
> at jcifs.smb.SmbFile.queryPath(SmbFile.java:770) ~[?:?]
> at jcifs.smb.SmbFile.exists(SmbFile.java:851) ~[?:?]
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2188)
> ~[?:?]
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:610)
> ~[?:?]
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> ~[mcf-pull-agent.jar:?]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1629) Support Solr Kerberos Authentication
[ https://issues.apache.org/jira/browse/CONNECTORS-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022122#comment-17022122 ] Karl Wright commented on CONNECTORS-1629: - Hi [[~jornfranke], can you include the URL of the pull request here? Thanks! > Support Solr Kerberos Authentication > > > Key: CONNECTORS-1629 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1629 > Project: ManifoldCF > Issue Type: Improvement > Components: Solr 7.x component >Affects Versions: ManifoldCF 2.14 >Reporter: Jörn Franke >Assignee: Karl Wright >Priority: Major > > Several enterprise deployments of Solr are leveraging SolrCloud Kerberos > authentication. > The integration seems to be rather simple and the goal of this Jira is to > evaluate the possential needed step to eventually contribute the Kerberos > integration to the ManifoldCF project. > The following steps would be needed: > * One can pass the JVM parameter java.security.auth.login.config to the > ManifoldCF JVM using -Djava.security.auth.login.config=/path/to/jaas.confg in > which Kerberos authentication details, such as keytab and principal that has > the right access to Solr is configured > * A small adaption to the SolrCloudClient that is used within Manifold needs > to be done to enable Kerberos authentication: > HttpClientUtil.setConfigurer(new Krb5HttpClientConfigurer()); > Should this be integrated in Manifold, one may want to consider one input > field in the configuration in the UI where one can select / flow which user > defined in the Jaas conf (you can define multiple one) should be chosen. By > default one may simply select "client" or "SolrJClient" if Jaas.conf is > present in the System properties. This does not mean the user needs to be > named like this, but the configuration entry referencing any user should be > named like this. > Having a confiugration allows to have a different users per flow. This might > also be needed in case you have multiple Solr clusters. > Related discussion > [http://mail-archives.apache.org/mod_mbox/manifoldcf-user/201912.mbox/browser] > SolrJ Kerberos integration: > [https://lucene.apache.org/solr/guide/8_3/kerberos-authentication-plugin.html#using-solrj-with-a-kerberized-solr] > Jaas conf documentation: > [https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CONNECTORS-1629) Support Solr Kerberos Authentication
[ https://issues.apache.org/jira/browse/CONNECTORS-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1629: --- Assignee: Karl Wright > Support Solr Kerberos Authentication > > > Key: CONNECTORS-1629 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1629 > Project: ManifoldCF > Issue Type: Improvement > Components: Solr 7.x component >Affects Versions: ManifoldCF 2.14 >Reporter: Jörn Franke >Assignee: Karl Wright >Priority: Major > > Several enterprise deployments of Solr are leveraging SolrCloud Kerberos > authentication. > The integration seems to be rather simple and the goal of this Jira is to > evaluate the possential needed step to eventually contribute the Kerberos > integration to the ManifoldCF project. > The following steps would be needed: > * One can pass the JVM parameter java.security.auth.login.config to the > ManifoldCF JVM using -Djava.security.auth.login.config=/path/to/jaas.confg in > which Kerberos authentication details, such as keytab and principal that has > the right access to Solr is configured > * A small adaption to the SolrCloudClient that is used within Manifold needs > to be done to enable Kerberos authentication: > HttpClientUtil.setConfigurer(new Krb5HttpClientConfigurer()); > Should this be integrated in Manifold, one may want to consider one input > field in the configuration in the UI where one can select / flow which user > defined in the Jaas conf (you can define multiple one) should be chosen. By > default one may simply select "client" or "SolrJClient" if Jaas.conf is > present in the System properties. This does not mean the user needs to be > named like this, but the configuration entry referencing any user should be > named like this. > Having a confiugration allows to have a different users per flow. This might > also be needed in case you have multiple Solr clusters. > Related discussion > [http://mail-archives.apache.org/mod_mbox/manifoldcf-user/201912.mbox/browser] > SolrJ Kerberos integration: > [https://lucene.apache.org/solr/guide/8_3/kerberos-authentication-plugin.html#using-solrj-with-a-kerberized-solr] > Jaas conf documentation: > [https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CONNECTORS-1631) Sharepoint connction problem
[ https://issues.apache.org/jira/browse/CONNECTORS-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1631. - Fix Version/s: ManifoldCF 2.15 Resolution: Fixed > Sharepoint connction problem > > > Key: CONNECTORS-1631 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1631 > Project: ManifoldCF > Issue Type: Task >Reporter: Zoltan Farago >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.15 > > Attachments: Manifold connection.png > > > Hello, > We are trying to connct to a Sharepoint 2016 site wich has default > installation. The URL is > [http://precogwin02/sites/UKAEAtestSP2016/_layouts/15/start.aspx#/Shared%20Documents/Forms/AllItems.aspx] > and from a browser it is fully operational. The site is installed on our > local network, no firewall issues could be. > When we try to connect from the Manifold CF we get this error message: "The > site at > [http://manifoldsharepoint/sites/UKAEAtestSP2016|http://manifoldsharepoint/sites/UKAEAtestSP2016/Shared%20Documents] > did not exist or was external; skipping" > This Manifold installation is able to connect to a Windows share on the same > server, so we think no user/pass Active Directory, etc issues could be here. > We checked forums, documentations but found no solution. > > Is there any special setting needed in Manifold, Sharepoint, et.? > > Thank you in advance! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1631) Sharepoint connction problem
[ https://issues.apache.org/jira/browse/CONNECTORS-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016840#comment-17016840 ] Karl Wright commented on CONNECTORS-1631: - Don't know about SharePoint Online. For SharePoint 2019 a plugin needs to be released that's properly linked against the SharePoint dll. If you are intending to use 2019 please let me know if you can supply the DLL so that we can set up and release the plugin. > Sharepoint connction problem > > > Key: CONNECTORS-1631 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1631 > Project: ManifoldCF > Issue Type: Task >Reporter: Zoltan Farago >Assignee: Karl Wright >Priority: Major > Attachments: Manifold connection.png > > > Hello, > We are trying to connct to a Sharepoint 2016 site wich has default > installation. The URL is > [http://precogwin02/sites/UKAEAtestSP2016/_layouts/15/start.aspx#/Shared%20Documents/Forms/AllItems.aspx] > and from a browser it is fully operational. The site is installed on our > local network, no firewall issues could be. > When we try to connect from the Manifold CF we get this error message: "The > site at > [http://manifoldsharepoint/sites/UKAEAtestSP2016|http://manifoldsharepoint/sites/UKAEAtestSP2016/Shared%20Documents] > did not exist or was external; skipping" > This Manifold installation is able to connect to a Windows share on the same > server, so we think no user/pass Active Directory, etc issues could be here. > We checked forums, documentations but found no solution. > > Is there any special setting needed in Manifold, Sharepoint, et.? > > Thank you in advance! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CONNECTORS-1632) Deprecate SSL and use TLS socket factories everywhere instead
Karl Wright created CONNECTORS-1632: --- Summary: Deprecate SSL and use TLS socket factories everywhere instead Key: CONNECTORS-1632 URL: https://issues.apache.org/jira/browse/CONNECTORS-1632 Project: ManifoldCF Issue Type: Task Components: Framework core, Lucene/SOLR connector Reporter: Karl Wright Assignee: Karl Wright Servers that serve only TLS apparently no longer work with ManifoldCF's various connectors. Changing the socket factory so that it supports the more modern protocols seems indicated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1631) Sharepoint connction problem
[ https://issues.apache.org/jira/browse/CONNECTORS-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016350#comment-17016350 ] Karl Wright commented on CONNECTORS-1631: - So did you install the MCF plugin for Sharepoint 2016 on the SharePoint server? If not, remember that this is mandatory. > Sharepoint connction problem > > > Key: CONNECTORS-1631 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1631 > Project: ManifoldCF > Issue Type: Task >Reporter: Zoltan Farago >Assignee: Karl Wright >Priority: Major > Attachments: Manifold connection.png > > > Hello, > We are trying to connct to a Sharepoint 2016 site wich has default > installation. The URL is > [http://precogwin02/sites/UKAEAtestSP2016/_layouts/15/start.aspx#/Shared%20Documents/Forms/AllItems.aspx] > and from a browser it is fully operational. The site is installed on our > local network, no firewall issues could be. > When we try to connect from the Manifold CF we get this error message: "The > site at > [http://manifoldsharepoint/sites/UKAEAtestSP2016|http://manifoldsharepoint/sites/UKAEAtestSP2016/Shared%20Documents] > did not exist or was external; skipping" > This Manifold installation is able to connect to a Windows share on the same > server, so we think no user/pass Active Directory, etc issues could be here. > We checked forums, documentations but found no solution. > > Is there any special setting needed in Manifold, Sharepoint, et.? > > Thank you in advance! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CONNECTORS-1631) Sharepoint connction problem
[ https://issues.apache.org/jira/browse/CONNECTORS-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1631: --- Assignee: Karl Wright > Sharepoint connction problem > > > Key: CONNECTORS-1631 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1631 > Project: ManifoldCF > Issue Type: Task >Reporter: Zoltan Farago >Assignee: Karl Wright >Priority: Major > Attachments: Manifold connection.png > > > Hello, > We are trying to connct to a Sharepoint 2016 site wich has default > installation. The URL is > [http://precogwin02/sites/UKAEAtestSP2016/_layouts/15/start.aspx#/Shared%20Documents/Forms/AllItems.aspx] > and from a browser it is fully operational. The site is installed on our > local network, no firewall issues could be. > When we try to connect from the Manifold CF we get this error message: "The > site at > [http://manifoldsharepoint/sites/UKAEAtestSP2016|http://manifoldsharepoint/sites/UKAEAtestSP2016/Shared%20Documents] > did not exist or was external; skipping" > This Manifold installation is able to connect to a Windows share on the same > server, so we think no user/pass Active Directory, etc issues could be here. > We checked forums, documentations but found no solution. > > Is there any special setting needed in Manifold, Sharepoint, et.? > > Thank you in advance! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1629) Support Solr Kerberos Authentication
[
https://issues.apache.org/jira/browse/CONNECTORS-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007319#comment-17007319
]
Karl Wright commented on CONNECTORS-1629:
-
Hi,
{quote}
About the ModifiedSolrClient - do I understand you correctly that you would
prefer to make the ModifiedSolrClient working in this setting as well? Ie by
creating a new ModifiedSolrClientKerberos and ModifiedLBSolrClientKerberos (not
touching the ones already in Manifold)? I can look at this, but I wonder if
this would still be needed as I did not observe any errors. Maybe the multipart
bit is fixed in higher Solr versions?
{quote}
I wish the multipart code was fixed but I fear it is not; I tried to get the
HttpClient team to agree to it but there was disagreement and I didn't get past
that. It's so long ago now that I don't even remember the discussion well, but
some team members thought that it was not the client's responsibility to
properly escape argument names when they were encoded in some cases but not in
others. If you are including metadata names and values that would require
encoding and this is working OK, then maybe this was resolved. But we should
evaluate that independently.
The multipart fix was only PART of the reason for ModifiedSolrHttpClient,
however. The other reason was that the Solr team essentially deprecated and
removed support for multipart posts entirely, which meant that streaming of
large documents to solr was not possible. I've kept that working and called
for them to rethink that problem, at which point I was told that nobody should
be using Solr Cell at all (!) So that stays until the Solr team figures this
out. The conversation there was at least relatively recent.
A github pull is fine. A diff gets generated by attaching a ".diff" to the URL
and then I can patch in svn.
> Support Solr Kerberos Authentication
>
>
> Key: CONNECTORS-1629
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1629
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Solr 7.x component
>Affects Versions: ManifoldCF 2.14
>Reporter: Jörn Franke
>Priority: Major
>
> Several enterprise deployments of Solr are leveraging SolrCloud Kerberos
> authentication.
> The integration seems to be rather simple and the goal of this Jira is to
> evaluate the possential needed step to eventually contribute the Kerberos
> integration to the ManifoldCF project.
> The following steps would be needed:
> * One can pass the JVM parameter java.security.auth.login.config to the
> ManifoldCF JVM using -Djava.security.auth.login.config=/path/to/jaas.confg in
> which Kerberos authentication details, such as keytab and principal that has
> the right access to Solr is configured
> * A small adaption to the SolrCloudClient that is used within Manifold needs
> to be done to enable Kerberos authentication:
> HttpClientUtil.setConfigurer(new Krb5HttpClientConfigurer());
> Should this be integrated in Manifold, one may want to consider one input
> field in the configuration in the UI where one can select / flow which user
> defined in the Jaas conf (you can define multiple one) should be chosen. By
> default one may simply select "client" or "SolrJClient" if Jaas.conf is
> present in the System properties. This does not mean the user needs to be
> named like this, but the configuration entry referencing any user should be
> named like this.
> Having a confiugration allows to have a different users per flow. This might
> also be needed in case you have multiple Solr clusters.
> Related discussion
> [http://mail-archives.apache.org/mod_mbox/manifoldcf-user/201912.mbox/browser]
> SolrJ Kerberos integration:
> [https://lucene.apache.org/solr/guide/8_3/kerberos-authentication-plugin.html#using-solrj-with-a-kerberized-solr]
> Jaas conf documentation:
> [https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1629) Support Solr Kerberos Authentication
[
https://issues.apache.org/jira/browse/CONNECTORS-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007086#comment-17007086
]
Karl Wright commented on CONNECTORS-1629:
-
Hi,
I suggest we make changes piecemeal. First, updating the Jetty version, and
the jars that are included, as described here:
{quote}
You need jetty-client-9.4.25.v20191220.jar (maybe a slightly older 9.4.x
version will do as well, the current manifold version not). Reason is that you
will get otherwise a java.lang.ClassNotFoundException:
org.eclipse.jetty.client.util.SPNEGOAuthentication error.
I was not exactly sure how to add this jar to the finally generated
distribution of ManifoldCF so i copied it in collector-lib and added it to the
classpath.
{quote}
To do this, we'd want to update the version of jetty specified in build.xml and
pom.xml, and add the new jar to the jetty jar list in build.xml. Then, in
framework/build.xml, the new jar should be added wherever jetty jars are found.
{quote}
I had to also deactivate the ModifiedLbSolrClient (commented out below)
otherwise you get an auth error 401. I believe the reason is that the default
SPNEGO Protocol for HTTP Kerberos always returns 401 not auth and THEN you are
supposed to do the Kerberos authentication, which is what SolrJ does
{quote}
The modified client is present because we need to be sure that the correct
(overridden) version of the SolrHttpClient class is used, not the default one.
So in this case you'd want to create a fresh copy of LBSolrClient and modify it
accordingly.
{quote}
Finally, you need to add to options.env.unix or options.env.win:
-Djava.security.auth.login.config=/path/to/jaas-client.conf
{quote}
I would suggest adding both the config file and the -D switch to all the
examples, but leave kerberos disabled unless somebody modifies the
jaas-client.conf file.
> Support Solr Kerberos Authentication
>
>
> Key: CONNECTORS-1629
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1629
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Solr 7.x component
>Affects Versions: ManifoldCF 2.14
>Reporter: Jörn Franke
>Priority: Major
>
> Several enterprise deployments of Solr are leveraging SolrCloud Kerberos
> authentication.
> The integration seems to be rather simple and the goal of this Jira is to
> evaluate the possential needed step to eventually contribute the Kerberos
> integration to the ManifoldCF project.
> The following steps would be needed:
> * One can pass the JVM parameter java.security.auth.login.config to the
> ManifoldCF JVM using -Djava.security.auth.login.config=/path/to/jaas.confg in
> which Kerberos authentication details, such as keytab and principal that has
> the right access to Solr is configured
> * A small adaption to the SolrCloudClient that is used within Manifold needs
> to be done to enable Kerberos authentication:
> HttpClientUtil.setConfigurer(new Krb5HttpClientConfigurer());
> Should this be integrated in Manifold, one may want to consider one input
> field in the configuration in the UI where one can select / flow which user
> defined in the Jaas conf (you can define multiple one) should be chosen. By
> default one may simply select "client" or "SolrJClient" if Jaas.conf is
> present in the System properties. This does not mean the user needs to be
> named like this, but the configuration entry referencing any user should be
> named like this.
> Having a confiugration allows to have a different users per flow. This might
> also be needed in case you have multiple Solr clusters.
> Related discussion
> [http://mail-archives.apache.org/mod_mbox/manifoldcf-user/201912.mbox/browser]
> SolrJ Kerberos integration:
> [https://lucene.apache.org/solr/guide/8_3/kerberos-authentication-plugin.html#using-solrj-with-a-kerberized-solr]
> Jaas conf documentation:
> [https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CONNECTORS-1630) Livelink/Opentext connector support REST API
[ https://issues.apache.org/jira/browse/CONNECTORS-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1630. - Fix Version/s: ManifoldCF 2.14 Resolution: Fixed The "csws" connector is the OpenText REST connector. It shipped with 2.14. > Livelink/Opentext connector support REST API > > > Key: CONNECTORS-1630 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1630 > Project: ManifoldCF > Issue Type: New Feature > Components: LiveLink connector >Reporter: Jörn Franke >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.14 > > > Currently, the Livelink connector is based on the Opentext proprietary APIs > lapi.jar/lssl.jar > It seems that Opentext/Livelink focuses most of their efforts on the public > REST API and lapi.jar becomes deprecated. Hence, a new connector shoule be > developed to leverage the REST API. > This task needs to investigate the minimum REST API version needed to provide > the Manifold functionality (Repository/Authority connection) similar to the > proprietary APIs. > One needs then also to identify the configuration options in the UI, such as > authority connection > * API base Url > * username/password auth (it is not basic auth), NTLM, Kerberos > * > repository: > * API base url > * API version to use (currently v1 or v2, just in case both version would > provide the needed functionality) > * username/password auth (it is not basic auth), NTLM, Kerberos > * path to fetch (e.g. by object id of the folder) > * recursive fetch (yes/no) > * regex pattern for specific filenames > * regex pattern for specific (sub-)folders in case of recursive fetch > * mapping of username to Livelink username > * number of threads for API calls > Then a plan needs to be developed on how to design the functionality. > Multi-threading should be used as much as possible, but should be limited to > a certain number of threads, e.g. by using a Execution Service, as the REST > API requires many calls to get all information (e.g. to get document > categories one needs to "work recursively its way up"). > > References: > * OpenText REST APIs Content server: > [https://developer.opentext.com/webaccess/#url=%2Fawd%2Fresources%2Fapis%2Fcs-rest-api-for-cs-16-s=501] > * OpenText REST API Directory services (this MIGHT be needed for the > Authority plugin, but it MAY also be fine just with the content server APIs): > [https://developer.opentext.com/webaccess/#url=%2Fawd%2Fresources%2Fapis%2Fotds-16=501] > * Executor service fixed thread pool: > [https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Executors.html#newFixedThreadPool(int]) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CONNECTORS-1630) Livelink/Opentext connector support REST API
[ https://issues.apache.org/jira/browse/CONNECTORS-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1630: --- Assignee: Karl Wright > Livelink/Opentext connector support REST API > > > Key: CONNECTORS-1630 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1630 > Project: ManifoldCF > Issue Type: New Feature > Components: LiveLink connector >Reporter: Jörn Franke >Assignee: Karl Wright >Priority: Major > > Currently, the Livelink connector is based on the Opentext proprietary APIs > lapi.jar/lssl.jar > It seems that Opentext/Livelink focuses most of their efforts on the public > REST API and lapi.jar becomes deprecated. Hence, a new connector shoule be > developed to leverage the REST API. > This task needs to investigate the minimum REST API version needed to provide > the Manifold functionality (Repository/Authority connection) similar to the > proprietary APIs. > One needs then also to identify the configuration options in the UI, such as > authority connection > * API base Url > * username/password auth (it is not basic auth), NTLM, Kerberos > * > repository: > * API base url > * API version to use (currently v1 or v2, just in case both version would > provide the needed functionality) > * username/password auth (it is not basic auth), NTLM, Kerberos > * path to fetch (e.g. by object id of the folder) > * recursive fetch (yes/no) > * regex pattern for specific filenames > * regex pattern for specific (sub-)folders in case of recursive fetch > * mapping of username to Livelink username > * number of threads for API calls > Then a plan needs to be developed on how to design the functionality. > Multi-threading should be used as much as possible, but should be limited to > a certain number of threads, e.g. by using a Execution Service, as the REST > API requires many calls to get all information (e.g. to get document > categories one needs to "work recursively its way up"). > > References: > * OpenText REST APIs Content server: > [https://developer.opentext.com/webaccess/#url=%2Fawd%2Fresources%2Fapis%2Fcs-rest-api-for-cs-16-s=501] > * OpenText REST API Directory services (this MIGHT be needed for the > Authority plugin, but it MAY also be fine just with the content server APIs): > [https://developer.opentext.com/webaccess/#url=%2Fawd%2Fresources%2Fapis%2Fotds-16=501] > * Executor service fixed thread pool: > [https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Executors.html#newFixedThreadPool(int]) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1586) Create plugin for Solr 8.0.0 when available
[ https://issues.apache.org/jira/browse/CONNECTORS-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998823#comment-16998823 ] Karl Wright commented on CONNECTORS-1586: - There's already a plugin release created, and I'd like to get it released before end of year. It also has to be compatible back to 8.0.0. See: https://svn.apache.org/repos/asf/manifoldcf/integration/solr-8.x/trunk > Create plugin for Solr 8.0.0 when available > --- > > Key: CONNECTORS-1586 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1586 > Project: ManifoldCF > Issue Type: Task >Reporter: Shinichiro Abe >Assignee: Karl Wright >Priority: Minor > Attachments: CONNECTORS-1568.patch > > > The plugin for Solr 8.0 release. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CONNECTORS-1586) Create plugin for Solr 8.0.0 when available
[ https://issues.apache.org/jira/browse/CONNECTORS-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1586: --- Assignee: Karl Wright > Create plugin for Solr 8.0.0 when available > --- > > Key: CONNECTORS-1586 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1586 > Project: ManifoldCF > Issue Type: Task >Reporter: Shinichiro Abe >Assignee: Karl Wright >Priority: Minor > Attachments: CONNECTORS-1568.patch > > > The plugin for Solr 8.0 release. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1628) Confluence Connector hang on error
[ https://issues.apache.org/jira/browse/CONNECTORS-1628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976686#comment-16976686 ] Karl Wright commented on CONNECTORS-1628: - Hi [~julienFL], this looks fine, please go ahead and commit to trunk. > Confluence Connector hang on error > -- > > Key: CONNECTORS-1628 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1628 > Project: ManifoldCF > Issue Type: Bug > Components: Confluence connector >Affects Versions: ManifoldCF 2.14 >Reporter: Julien Massiera >Assignee: Julien Massiera >Priority: Critical > Fix For: ManifoldCF 2.15 > > Attachments: CONNECTORS-1628.diff > > > During a crawling job, if the confluence connector encounters error(s) on > requests, it hang and there is no other way than restarting the MCF agent so > it works again. > The reason is that the connector does not release the HTTP response if an > exception or an HTTP error is encountered during its processing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CONNECTORS-1628) Confluence Connector hang on error
[ https://issues.apache.org/jira/browse/CONNECTORS-1628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1628: --- Assignee: Julien Massiera > Confluence Connector hang on error > -- > > Key: CONNECTORS-1628 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1628 > Project: ManifoldCF > Issue Type: Bug > Components: Confluence connector >Affects Versions: ManifoldCF 2.14 >Reporter: Julien Massiera >Assignee: Julien Massiera >Priority: Critical > Fix For: ManifoldCF 2.15 > > Attachments: CONNECTORS-1628.diff > > > During a crawling job, if the confluence connector encounters error(s) on > requests, it hang and there is no other way than restarting the MCF agent so > it works again. > The reason is that the connector does not release the HTTP response if an > exception or an HTTP error is encountered during its processing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CONNECTORS-1627) CSWS Connector: Error tossed: null (ownerRights may be null)
[
https://issues.apache.org/jira/browse/CONNECTORS-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1627.
-
Fix Version/s: ManifoldCF 2.15
Assignee: Karl Wright
Resolution: Fixed
r1868956
> CSWS Connector: Error tossed: null (ownerRights may be null)
>
>
> Key: CONNECTORS-1627
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1627
> Project: ManifoldCF
> Issue Type: Bug
> Components: LiveLink connector
>Reporter: Markus Schuch
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.15
>
> Attachments: CONNECTORS-1627.patch, screenshot-1.png
>
>
> We encounter documents having object rights with {{ownerRights}} = {{null}}
> leading to:
> {code}
> FATAL 2019-10-25T10:55:03,839 (Worker thread '15') - Error tossed: null
> java.lang.NullPointerException
> at
> org.apache.manifoldcf.crawler.connectors.csws.CswsConnector.processDocuments(CswsConnector.java:1276)
> ~[?:?]
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CONNECTORS-1626) CSWS Authority does no return all user permissions
[ https://issues.apache.org/jira/browse/CONNECTORS-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1626: --- Assignee: Markus Schuch > CSWS Authority does no return all user permissions > -- > > Key: CONNECTORS-1626 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1626 > Project: ManifoldCF > Issue Type: Bug > Components: LiveLink connector >Reporter: Markus Schuch >Assignee: Markus Schuch >Priority: Major > > Currently the CSWS Authority does return tokens for groups, a user is a > directly a member of. > The CSWS Authority does not return > - tokens for transitive group memberships > - tokens for project group memberships -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CONNECTORS-1117) Create a livelink connector using the LiveLink REST API
[ https://issues.apache.org/jira/browse/CONNECTORS-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1117. - Fix Version/s: (was: ManifoldCF next) Resolution: Duplicate > Create a livelink connector using the LiveLink REST API > --- > > Key: CONNECTORS-1117 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1117 > Project: ManifoldCF > Issue Type: Improvement > Components: LiveLink connector >Reporter: Karl Wright >Assignee: Kishore Kumar >Priority: Major > Labels: gsoc2015, gsoc2016 > > LAPI is deprecated, so develop a connector that doesn't use it. > Here's API documentation: > https://developer.opentext.com/awd/forums/questions/1130441#r1130448 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1625) When processing a specific PDF Manifold goes out of memory
[ https://issues.apache.org/jira/browse/CONNECTORS-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944386#comment-16944386 ] Karl Wright commented on CONNECTORS-1625: - Also, FWIW, the default Java memory sizes on the example are not guaranteed to allow processing of N simultaneous Tika extractions (one per worker thread) of the sort that require more memory. Memory sizes allocated to the JVM are settable in the start-options files, and the first thing you want to do is increase those values to see if the problem goes away for you. > When processing a specific PDF Manifold goes out of memory > -- > > Key: CONNECTORS-1625 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1625 > Project: ManifoldCF > Issue Type: Bug > Components: Tika extractor >Affects Versions: ManifoldCF 2.12 >Reporter: Donald Van den Driessche >Assignee: Karl Wright >Priority: Major > Attachments: abd-serotec-antibodies-uk.pdf > > > When processing attached file with manifoldcf 2.12, we keep getting an out of > memory error. > When just parsing it throug Tika 1.18, no issues are being found. > Can anyone look into it? > Thanks in advance! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CONNECTORS-1625) When processing a specific PDF Manifold goes out of memory
[ https://issues.apache.org/jira/browse/CONNECTORS-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944385#comment-16944385 ] Karl Wright edited comment on CONNECTORS-1625 at 10/4/19 10:29 AM: --- What version of Manifold is this? 2.12 is pretty old by Tika standards. We pretty much upgrade Tika continuously at this point and if it's not the current version you are running old Tika code. was (Author: kwri...@metacarta.com): What version of Manifold is this? We pretty much upgrade Tika continuously at this point and if it's not the current version you are running old Tika code. > When processing a specific PDF Manifold goes out of memory > -- > > Key: CONNECTORS-1625 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1625 > Project: ManifoldCF > Issue Type: Bug > Components: Tika extractor >Affects Versions: ManifoldCF 2.12 >Reporter: Donald Van den Driessche >Assignee: Karl Wright >Priority: Major > Attachments: abd-serotec-antibodies-uk.pdf > > > When processing attached file with manifoldcf 2.12, we keep getting an out of > memory error. > When just parsing it throug Tika 1.18, no issues are being found. > Can anyone look into it? > Thanks in advance! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CONNECTORS-1625) When processing a specific PDF Manifold goes out of memory
[ https://issues.apache.org/jira/browse/CONNECTORS-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-1625: --- Assignee: Karl Wright > When processing a specific PDF Manifold goes out of memory > -- > > Key: CONNECTORS-1625 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1625 > Project: ManifoldCF > Issue Type: Bug > Components: Tika extractor >Affects Versions: ManifoldCF 2.12 >Reporter: Donald Van den Driessche >Assignee: Karl Wright >Priority: Major > Attachments: abd-serotec-antibodies-uk.pdf > > > When processing attached file with manifoldcf 2.12, we keep getting an out of > memory error. > When just parsing it throug Tika 1.18, no issues are being found. > Can anyone look into it? > Thanks in advance! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CONNECTORS-1624) Get ManifoldCF to run under Java 11 or higher
Karl Wright created CONNECTORS-1624: --- Summary: Get ManifoldCF to run under Java 11 or higher Key: CONNECTORS-1624 URL: https://issues.apache.org/jira/browse/CONNECTORS-1624 Project: ManifoldCF Issue Type: Task Components: Framework core Reporter: Karl Wright Assignee: Karl Wright Fix For: ManifoldCF 2.15 Java 11 doesn't include a number of classes that Java 8 does. We need to explicitly include jars that provide these classes or ManifoldCF will not function under higher Java revs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CONNECTORS-1623) Script tags not ignored
[
https://issues.apache.org/jira/browse/CONNECTORS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-1623.
-
Resolution: Fixed
> Script tags not ignored
> ---
>
> Key: CONNECTORS-1623
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1623
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
>Affects Versions: ManifoldCF 2.13
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
> Fix For: ManifoldCF 2.14
>
>
> I discovered a problematic behavior with the
> org.apache.manifoldcf.connectorcommon.fuzzyml.TagParseState class when
> crawling web pages. This behavior poses problem in particular for the
> scenario of form based authentication, as explained further.
> The org.apache.manifoldcf.connectorcommon.fuzzyml.HTMLParseState class which
> is called by the TagParseState on each noteTag() or noteEndTag() methods,
> uses the org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState
> class to detect if the parsing process is in or out of a 'script' tag and
> then do something or not with the incoming data. The problem is that the
> TagParseState class is not aware of the type of tag currently parsed, so it
> continues to analyze any char encountered to detect tags even if it is
> actually parsing a script tag.
> So let's imagine you have a script tag built like this in a web page:
> {code:java}
> if(myvar <= 9) {...}
> {code}
> When the TagParseState parses the char '<' it will consider that a new tag
> begins until it encounters a '>' char. So in the case above, the
> TagParseState will never catch the end of the script tag, and thus, the
> scriptParseState variable in the ScriptParseState class will remain in the
> SCRIPTPARSESTATE_INSCRIPT state and the rest of the web page will not be
> correctly handled by the other parsers.
> As a result, if you, for example, configure a form authentication for your
> crawl and that the form web page contains this kind of script tag prior to
> the form tag, the form will never be handled and the authentication will
> fail. This was the case I encountered, and I resolved it by forcing the
> scriptParseState to be SCRIPTPARSESTATE_NORMAL.
> ref :
> [http://mail-archives.apache.org/mod_mbox/manifoldcf-dev/201909.mbox/%3CCALUFAGA7eXi_gNBqWv2PRt2FaXuuKW5rTwLiXfceTkUAQfBvVg%40mail.gmail.com%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1623) Script tags not ignored
[
https://issues.apache.org/jira/browse/CONNECTORS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936967#comment-16936967
]
Karl Wright commented on CONNECTORS-1623:
-
r1867468 (trunk)
r1867469 (release branch)
> Script tags not ignored
> ---
>
> Key: CONNECTORS-1623
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1623
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
>Affects Versions: ManifoldCF 2.13
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
> Fix For: ManifoldCF 2.14
>
>
> I discovered a problematic behavior with the
> org.apache.manifoldcf.connectorcommon.fuzzyml.TagParseState class when
> crawling web pages. This behavior poses problem in particular for the
> scenario of form based authentication, as explained further.
> The org.apache.manifoldcf.connectorcommon.fuzzyml.HTMLParseState class which
> is called by the TagParseState on each noteTag() or noteEndTag() methods,
> uses the org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState
> class to detect if the parsing process is in or out of a 'script' tag and
> then do something or not with the incoming data. The problem is that the
> TagParseState class is not aware of the type of tag currently parsed, so it
> continues to analyze any char encountered to detect tags even if it is
> actually parsing a script tag.
> So let's imagine you have a script tag built like this in a web page:
> {code:java}
> if(myvar <= 9) {...}
> {code}
> When the TagParseState parses the char '<' it will consider that a new tag
> begins until it encounters a '>' char. So in the case above, the
> TagParseState will never catch the end of the script tag, and thus, the
> scriptParseState variable in the ScriptParseState class will remain in the
> SCRIPTPARSESTATE_INSCRIPT state and the rest of the web page will not be
> correctly handled by the other parsers.
> As a result, if you, for example, configure a form authentication for your
> crawl and that the form web page contains this kind of script tag prior to
> the form tag, the form will never be handled and the authentication will
> fail. This was the case I encountered, and I resolved it by forcing the
> scriptParseState to be SCRIPTPARSESTATE_NORMAL.
> ref :
> [http://mail-archives.apache.org/mod_mbox/manifoldcf-dev/201909.mbox/%3CCALUFAGA7eXi_gNBqWv2PRt2FaXuuKW5rTwLiXfceTkUAQfBvVg%40mail.gmail.com%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1623) Script tags not ignored
[
https://issues.apache.org/jira/browse/CONNECTORS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936964#comment-16936964
]
Karl Wright commented on CONNECTORS-1623:
-
I found a fix; committing it.
> Script tags not ignored
> ---
>
> Key: CONNECTORS-1623
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1623
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
>Affects Versions: ManifoldCF 2.13
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
> Fix For: ManifoldCF 2.14
>
>
> I discovered a problematic behavior with the
> org.apache.manifoldcf.connectorcommon.fuzzyml.TagParseState class when
> crawling web pages. This behavior poses problem in particular for the
> scenario of form based authentication, as explained further.
> The org.apache.manifoldcf.connectorcommon.fuzzyml.HTMLParseState class which
> is called by the TagParseState on each noteTag() or noteEndTag() methods,
> uses the org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState
> class to detect if the parsing process is in or out of a 'script' tag and
> then do something or not with the incoming data. The problem is that the
> TagParseState class is not aware of the type of tag currently parsed, so it
> continues to analyze any char encountered to detect tags even if it is
> actually parsing a script tag.
> So let's imagine you have a script tag built like this in a web page:
> {code:java}
> if(myvar <= 9) {...}
> {code}
> When the TagParseState parses the char '<' it will consider that a new tag
> begins until it encounters a '>' char. So in the case above, the
> TagParseState will never catch the end of the script tag, and thus, the
> scriptParseState variable in the ScriptParseState class will remain in the
> SCRIPTPARSESTATE_INSCRIPT state and the rest of the web page will not be
> correctly handled by the other parsers.
> As a result, if you, for example, configure a form authentication for your
> crawl and that the form web page contains this kind of script tag prior to
> the form tag, the form will never be handled and the authentication will
> fail. This was the case I encountered, and I resolved it by forcing the
> scriptParseState to be SCRIPTPARSESTATE_NORMAL.
> ref :
> [http://mail-archives.apache.org/mod_mbox/manifoldcf-dev/201909.mbox/%3CCALUFAGA7eXi_gNBqWv2PRt2FaXuuKW5rTwLiXfceTkUAQfBvVg%40mail.gmail.com%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1623) Script tags not ignored
[
https://issues.apache.org/jira/browse/CONNECTORS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936934#comment-16936934
]
Karl Wright commented on CONNECTORS-1623:
-
Verification failed, with this unit test failure:
{code}
run-connector-common-tests:
[junit] Testsuite: org.apache.manifoldcf.connectorcommon.fuzzyml.TestFuzzyML
[junit] ERROR StatusLogger No log4j2 configuration file found. Using
default configuration: logging only errors to the console.
[junit] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
0.344 sec
[junit]
[junit] - Standard Error -
[junit] ERROR StatusLogger No log4j2 configuration file found. Using
default configuration: logging only errors to the console.
[junit] - ---
[junit] Testcase:
testTags(org.apache.manifoldcf.connectorcommon.fuzzyml.TestFuzzyML): FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at
org.apache.manifoldcf.connectorcommon.fuzzyml.TestFuzzyML.testTags(TestFuzzyML.java:192)
[junit]
[junit]
BUILD FAILED
C:\wip\mcf\trunk\build.xml:290: The following error occurred while executing
this line:
C:\wip\mcf\trunk\framework\build.xml:2030: Test
org.apache.manifoldcf.connectorcommon.fuzzyml.TestFuzzyML failed
{code}
The test is using a real-world example HTML page and parsing it, and it fails
because it does not correctly pick up the tag at the end of the
script section. The reason may be that end tags are still processed within the
script section and that confuses the tag pairing. That will not be
straightforward to fix. [~julienFL], awaiting your suggestion for that.
> Script tags not ignored
> ---
>
> Key: CONNECTORS-1623
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1623
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
>Affects Versions: ManifoldCF 2.13
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
> Fix For: ManifoldCF 2.14
>
>
> I discovered a problematic behavior with the
> org.apache.manifoldcf.connectorcommon.fuzzyml.TagParseState class when
> crawling web pages. This behavior poses problem in particular for the
> scenario of form based authentication, as explained further.
> The org.apache.manifoldcf.connectorcommon.fuzzyml.HTMLParseState class which
> is called by the TagParseState on each noteTag() or noteEndTag() methods,
> uses the org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState
> class to detect if the parsing process is in or out of a 'script' tag and
> then do something or not with the incoming data. The problem is that the
> TagParseState class is not aware of the type of tag currently parsed, so it
> continues to analyze any char encountered to detect tags even if it is
> actually parsing a script tag.
> So let's imagine you have a script tag built like this in a web page:
> {code:java}
> if(myvar <= 9) {...}
> {code}
> When the TagParseState parses the char '<' it will consider that a new tag
> begins until it encounters a '>' char. So in the case above, the
> TagParseState will never catch the end of the script tag, and thus, the
> scriptParseState variable in the ScriptParseState class will remain in the
> SCRIPTPARSESTATE_INSCRIPT state and the rest of the web page will not be
> correctly handled by the other parsers.
> As a result, if you, for example, configure a form authentication for your
> crawl and that the form web page contains this kind of script tag prior to
> the form tag, the form will never be handled and the authentication will
> fail. This was the case I encountered, and I resolved it by forcing the
> scriptParseState to be SCRIPTPARSESTATE_NORMAL.
> ref :
> [http://mail-archives.apache.org/mod_mbox/manifoldcf-dev/201909.mbox/%3CCALUFAGA7eXi_gNBqWv2PRt2FaXuuKW5rTwLiXfceTkUAQfBvVg%40mail.gmail.com%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1623) Script tags not ignored
[
https://issues.apache.org/jira/browse/CONNECTORS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936866#comment-16936866
]
Karl Wright commented on CONNECTORS-1623:
-
I put together a fix but need to verify it.
> Script tags not ignored
> ---
>
> Key: CONNECTORS-1623
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1623
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
>Affects Versions: ManifoldCF 2.13
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
> Fix For: ManifoldCF 2.14
>
>
> I discovered a problematic behavior with the
> org.apache.manifoldcf.connectorcommon.fuzzyml.TagParseState class when
> crawling web pages. This behavior poses problem in particular for the
> scenario of form based authentication, as explained further.
> The org.apache.manifoldcf.connectorcommon.fuzzyml.HTMLParseState class which
> is called by the TagParseState on each noteTag() or noteEndTag() methods,
> uses the org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState
> class to detect if the parsing process is in or out of a 'script' tag and
> then do something or not with the incoming data. The problem is that the
> TagParseState class is not aware of the type of tag currently parsed, so it
> continues to analyze any char encountered to detect tags even if it is
> actually parsing a script tag.
> So let's imagine you have a script tag built like this in a web page:
> {code:java}
> if(myvar <= 9) {...}
> {code}
> When the TagParseState parses the char '<' it will consider that a new tag
> begins until it encounters a '>' char. So in the case above, the
> TagParseState will never catch the end of the script tag, and thus, the
> scriptParseState variable in the ScriptParseState class will remain in the
> SCRIPTPARSESTATE_INSCRIPT state and the rest of the web page will not be
> correctly handled by the other parsers.
> As a result, if you, for example, configure a form authentication for your
> crawl and that the form web page contains this kind of script tag prior to
> the form tag, the form will never be handled and the authentication will
> fail. This was the case I encountered, and I resolved it by forcing the
> scriptParseState to be SCRIPTPARSESTATE_NORMAL.
> ref :
> [http://mail-archives.apache.org/mod_mbox/manifoldcf-dev/201909.mbox/%3CCALUFAGA7eXi_gNBqWv2PRt2FaXuuKW5rTwLiXfceTkUAQfBvVg%40mail.gmail.com%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CONNECTORS-1623) Script tags not ignored
[
https://issues.apache.org/jira/browse/CONNECTORS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright updated CONNECTORS-1623:
Fix Version/s: ManifoldCF 2.14
> Script tags not ignored
> ---
>
> Key: CONNECTORS-1623
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1623
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
>Affects Versions: ManifoldCF 2.13
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
> Fix For: ManifoldCF 2.14
>
>
> I discovered a problematic behavior with the
> org.apache.manifoldcf.connectorcommon.fuzzyml.TagParseState class when
> crawling web pages. This behavior poses problem in particular for the
> scenario of form based authentication, as explained further.
> The org.apache.manifoldcf.connectorcommon.fuzzyml.HTMLParseState class which
> is called by the TagParseState on each noteTag() or noteEndTag() methods,
> uses the org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState
> class to detect if the parsing process is in or out of a 'script' tag and
> then do something or not with the incoming data. The problem is that the
> TagParseState class is not aware of the type of tag currently parsed, so it
> continues to analyze any char encountered to detect tags even if it is
> actually parsing a script tag.
> So let's imagine you have a script tag built like this in a web page:
> {code:java}
> if(myvar <= 9) {...}
> {code}
> When the TagParseState parses the char '<' it will consider that a new tag
> begins until it encounters a '>' char. So in the case above, the
> TagParseState will never catch the end of the script tag, and thus, the
> scriptParseState variable in the ScriptParseState class will remain in the
> SCRIPTPARSESTATE_INSCRIPT state and the rest of the web page will not be
> correctly handled by the other parsers.
> As a result, if you, for example, configure a form authentication for your
> crawl and that the form web page contains this kind of script tag prior to
> the form tag, the form will never be handled and the authentication will
> fail. This was the case I encountered, and I resolved it by forcing the
> scriptParseState to be SCRIPTPARSESTATE_NORMAL.
> ref :
> [http://mail-archives.apache.org/mod_mbox/manifoldcf-dev/201909.mbox/%3CCALUFAGA7eXi_gNBqWv2PRt2FaXuuKW5rTwLiXfceTkUAQfBvVg%40mail.gmail.com%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CONNECTORS-1623) Script tags not ignored
[
https://issues.apache.org/jira/browse/CONNECTORS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-1623:
---
Assignee: Karl Wright
> Script tags not ignored
> ---
>
> Key: CONNECTORS-1623
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1623
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
>Affects Versions: ManifoldCF 2.13
>Reporter: Julien Massiera
>Assignee: Karl Wright
>Priority: Critical
>
> I discovered a problematic behavior with the
> org.apache.manifoldcf.connectorcommon.fuzzyml.TagParseState class when
> crawling web pages. This behavior poses problem in particular for the
> scenario of form based authentication, as explained further.
> The org.apache.manifoldcf.connectorcommon.fuzzyml.HTMLParseState class which
> is called by the TagParseState on each noteTag() or noteEndTag() methods,
> uses the org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState
> class to detect if the parsing process is in or out of a 'script' tag and
> then do something or not with the incoming data. The problem is that the
> TagParseState class is not aware of the type of tag currently parsed, so it
> continues to analyze any char encountered to detect tags even if it is
> actually parsing a script tag.
> So let's imagine you have a script tag built like this in a web page:
> {code:java}
> if(myvar <= 9) {...}
> {code}
> When the TagParseState parses the char '<' it will consider that a new tag
> begins until it encounters a '>' char. So in the case above, the
> TagParseState will never catch the end of the script tag, and thus, the
> scriptParseState variable in the ScriptParseState class will remain in the
> SCRIPTPARSESTATE_INSCRIPT state and the rest of the web page will not be
> correctly handled by the other parsers.
> As a result, if you, for example, configure a form authentication for your
> crawl and that the form web page contains this kind of script tag prior to
> the form tag, the form will never be handled and the authentication will
> fail. This was the case I encountered, and I resolved it by forcing the
> scriptParseState to be SCRIPTPARSESTATE_NORMAL.
> ref :
> [http://mail-archives.apache.org/mod_mbox/manifoldcf-dev/201909.mbox/%3CCALUFAGA7eXi_gNBqWv2PRt2FaXuuKW5rTwLiXfceTkUAQfBvVg%40mail.gmail.com%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CONNECTORS-1566) Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-1566. - Resolution: Fixed Confirmed that the connector is mostly working for multiple users. We reserve the right to open individual tickets for problems that still need resolution. > Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector > -- > > Key: CONNECTORS-1566 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1566 > Project: ManifoldCF > Issue Type: Task > Components: LiveLink connector >Affects Versions: ManifoldCF 2.12 >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.14 > > Attachments: OTCS_IIS.png, OTCS_Tomcat.png, chrome_cgfC00ujx7.png > > > LAPI is being deprecated. We need to develop a replacement for it using the > ContentServer Web Services API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1566) Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936085#comment-16936085 ] Karl Wright commented on CONNECTORS-1566: - [~schuch], is the latest code working for you? If so, maybe we can spin the new release finally. > Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector > -- > > Key: CONNECTORS-1566 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1566 > Project: ManifoldCF > Issue Type: Task > Components: LiveLink connector >Affects Versions: ManifoldCF 2.12 >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.14 > > Attachments: OTCS_IIS.png, OTCS_Tomcat.png, chrome_cgfC00ujx7.png > > > LAPI is being deprecated. We need to develop a replacement for it using the > ContentServer Web Services API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1566) Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934487#comment-16934487 ] Karl Wright commented on CONNECTORS-1566: - [~schuch], yes, that was the semantic for LAPI; version 0 was the latest version. I believe this is supported in csws or you would not index any documents at all. But I can believe that there are documents that have no versions at all, and then you might get this error. > Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector > -- > > Key: CONNECTORS-1566 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1566 > Project: ManifoldCF > Issue Type: Task > Components: LiveLink connector >Affects Versions: ManifoldCF 2.12 >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.14 > > Attachments: OTCS_IIS.png, OTCS_Tomcat.png, chrome_cgfC00ujx7.png > > > LAPI is being deprecated. We need to develop a replacement for it using the > ContentServer Web Services API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1566) Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934473#comment-16934473 ] Karl Wright commented on CONNECTORS-1566: - [~schuch], I've updated the code to detect this condition and not fail. > Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector > -- > > Key: CONNECTORS-1566 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1566 > Project: ManifoldCF > Issue Type: Task > Components: LiveLink connector >Affects Versions: ManifoldCF 2.12 >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.14 > > Attachments: OTCS_IIS.png, OTCS_Tomcat.png, chrome_cgfC00ujx7.png > > > LAPI is being deprecated. We need to develop a replacement for it using the > ContentServer Web Services API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CONNECTORS-1566) Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933472#comment-16933472 ] Karl Wright edited comment on CONNECTORS-1566 at 9/19/19 7:53 PM: -- Have you looked up the error code? It sounds like we're trying to get the latest version of a document but it is not actually present? I need to know what getFaultCode() returns for this error. was (Author: kwri...@metacarta.com): Have you looked up the error code? It sounds like we're trying to get the latest version of a document but it is not actually present? > Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector > -- > > Key: CONNECTORS-1566 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1566 > Project: ManifoldCF > Issue Type: Task > Components: LiveLink connector >Affects Versions: ManifoldCF 2.12 >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.14 > > Attachments: OTCS_IIS.png, OTCS_Tomcat.png, chrome_cgfC00ujx7.png > > > LAPI is being deprecated. We need to develop a replacement for it using the > ContentServer Web Services API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1566) Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933472#comment-16933472 ] Karl Wright commented on CONNECTORS-1566: - Have you looked up the error code? It sounds like we're trying to get the latest version of a document but it is not actually present? > Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector > -- > > Key: CONNECTORS-1566 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1566 > Project: ManifoldCF > Issue Type: Task > Components: LiveLink connector >Affects Versions: ManifoldCF 2.12 >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.14 > > Attachments: OTCS_IIS.png, OTCS_Tomcat.png, chrome_cgfC00ujx7.png > > > LAPI is being deprecated. We need to develop a replacement for it using the > ContentServer Web Services API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1566) Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932322#comment-16932322 ] Karl Wright commented on CONNECTORS-1566: - I committed the naive patch. > Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector > -- > > Key: CONNECTORS-1566 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1566 > Project: ManifoldCF > Issue Type: Task > Components: LiveLink connector >Affects Versions: ManifoldCF 2.12 >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.14 > > Attachments: OTCS_IIS.png, OTCS_Tomcat.png, chrome_cgfC00ujx7.png > > > LAPI is being deprecated. We need to develop a replacement for it using the > ContentServer Web Services API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1566) Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932319#comment-16932319 ] Karl Wright commented on CONNECTORS-1566: - The naive patch seems as good as any to me. The question is under what conditions do you get back a null rights object? In any case, if it fails it would fail in a more restrictive way rather than a less restrictive. > Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector > -- > > Key: CONNECTORS-1566 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1566 > Project: ManifoldCF > Issue Type: Task > Components: LiveLink connector >Affects Versions: ManifoldCF 2.12 >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.14 > > Attachments: OTCS_IIS.png, OTCS_Tomcat.png, chrome_cgfC00ujx7.png > > > LAPI is being deprecated. We need to develop a replacement for it using the > ContentServer Web Services API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1566) Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931829#comment-16931829 ] Karl Wright commented on CONNECTORS-1566: - [~schuch] I added this suffix for all queries performed. Hopefully that will do the trick. Please let me know. > Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector > -- > > Key: CONNECTORS-1566 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1566 > Project: ManifoldCF > Issue Type: Task > Components: LiveLink connector >Affects Versions: ManifoldCF 2.12 >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.14 > > Attachments: OTCS_IIS.png, OTCS_Tomcat.png, chrome_cgfC00ujx7.png > > > LAPI is being deprecated. We need to develop a replacement for it using the > ContentServer Web Services API. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (CONNECTORS-1566) Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector
[
https://issues.apache.org/jira/browse/CONNECTORS-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930505#comment-16930505
]
Karl Wright edited comment on CONNECTORS-1566 at 9/16/19 12:43 PM:
---
[~schuch], I do not know of anything done to enable OTSubType; seems that that
field was in the doc. The other team DID mention needing to (somehow!) change
the parent ID to 2004, but I committed code that should have done that, I
thought. It's possible that the code needs to look up the CategoriesWS node as
the parent? I don't know but can you verify that 2004 does the trick?
The method that should be finding the category workspace from a category path
is this:
{code}
public RootValue(CswsContext llc, String pathString)
{code}
This peels the prefix off the beginning of the path and assumes it's the
workspace name. It then looks up object information for that workspace node,
and gets the ID, which in this case should be 2004 if my assumptions are
correct.
Can you verify that the path has the correct prefix, and that the ID returned
is 2004?
was (Author: kwri...@metacarta.com):
[~schuch], I do not know of anything done to enable OTSubType; seems that that
field was in the doc. The other team DID mention needing to (somehow!) change
the parent ID to 2004, but I committed code that should have done that, I
thought. It's possible that the code needs to look up the CategoriesWS node as
the parent? I don't know but can you verify that 2004 does the trick?
> Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector
> --
>
> Key: CONNECTORS-1566
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1566
> Project: ManifoldCF
> Issue Type: Task
> Components: LiveLink connector
>Affects Versions: ManifoldCF 2.12
>Reporter: Karl Wright
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.14
>
> Attachments: OTCS_IIS.png, OTCS_Tomcat.png, chrome_cgfC00ujx7.png
>
>
> LAPI is being deprecated. We need to develop a replacement for it using the
> ContentServer Web Services API.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (CONNECTORS-1566) Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930505#comment-16930505 ] Karl Wright commented on CONNECTORS-1566: - [~schuch], I do not know of anything done to enable OTSubType; seems that that field was in the doc. The other team DID mention needing to (somehow!) change the parent ID to 2004, but I committed code that should have done that, I thought. It's possible that the code needs to look up the CategoriesWS node as the parent? I don't know but can you verify that 2004 does the trick? > Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector > -- > > Key: CONNECTORS-1566 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1566 > Project: ManifoldCF > Issue Type: Task > Components: LiveLink connector >Affects Versions: ManifoldCF 2.12 >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.14 > > Attachments: OTCS_IIS.png, OTCS_Tomcat.png, chrome_cgfC00ujx7.png > > > LAPI is being deprecated. We need to develop a replacement for it using the > ContentServer Web Services API. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CONNECTORS-1566) Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector
[ https://issues.apache.org/jira/browse/CONNECTORS-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930418#comment-16930418 ] Karl Wright commented on CONNECTORS-1566: - [~schuch], the other team was able to see paths suggested. Can you put debugging statements enough to figure out what the problem is in your environment? > Develop CSWS connector as a replacement for deprecated LiveLink LAPI connector > -- > > Key: CONNECTORS-1566 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1566 > Project: ManifoldCF > Issue Type: Task > Components: LiveLink connector >Affects Versions: ManifoldCF 2.12 >Reporter: Karl Wright >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.14 > > Attachments: OTCS_IIS.png, OTCS_Tomcat.png, chrome_cgfC00ujx7.png > > > LAPI is being deprecated. We need to develop a replacement for it using the > ContentServer Web Services API. -- This message was sent by Atlassian Jira (v8.3.2#803003)
