[jira] [Issue Comment Edited] (CONNECTORS-263) Long-running tests should all be moved to Maven's integration-test phase
[ https://issues.apache.org/jira/browse/CONNECTORS-263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115827#comment-13115827 ] Karl Wright edited comment on CONNECTORS-263 at 9/27/11 7:30 PM: - There is a different ticket for integrating maven profiles with the build. CONNECTORS-261. By default, the maven build should not run anything other than the general, Derby and HSQLDB tests. The PostgreSQL tests should not be run under Maven. So if this is happening it is a bug. was (Author: kwri...@metacarta.com): There is a different ticket for integrating maven profiles with the build. CONNECTORS-261. > Long-running tests should all be moved to Maven's integration-test phase > > > Key: CONNECTORS-263 > URL: https://issues.apache.org/jira/browse/CONNECTORS-263 > Project: ManifoldCF > Issue Type: Improvement > Components: Build >Affects Versions: ManifoldCF 0.4 >Reporter: Karl Wright >Assignee: Piergiorgio Lucidi > Fix For: ManifoldCF 0.4 > > > Long-running tests should be moved to Maven's integration-test phase. > (which are all in root-level "tests" directory) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-256) Connector for crawling Wikis
[ https://issues.apache.org/jira/browse/CONNECTORS-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127534#comment-13127534 ] Karl Wright edited comment on CONNECTORS-256 at 10/14/11 1:30 PM: -- bq. Also, information about the last_modified date was missing completely. The last_modified date I did not include in the metadata indexed; this is why I wanted other folks to try it out. I've opened a new ticket to capture that enhancement: CONNECTORS-273. was (Author: kwri...@metacarta.com): bq. Also, information about the last_modified date was missing completely. The last_modified date I did not include in the metadata indexed; this is why I wanted other folks to try it out. I'll open a new ticket to capture that enhancement. > Connector for crawling Wikis > > > Key: CONNECTORS-256 > URL: https://issues.apache.org/jira/browse/CONNECTORS-256 > Project: ManifoldCF > Issue Type: New Feature > Components: Wiki connector >Affects Versions: ManifoldCF 0.4 >Reporter: Karl Wright >Assignee: Karl Wright > Fix For: ManifoldCF 0.4 > > > People have been trying to crawl wikis with ManifoldCF, but using the generic > crawler is not a good way to do this. Instead, it looks like we really could > use a wiki connector, which would understand the wiki API and thus crawl wiki > content quickly and effectively. > Some pertinent API references follow: > I don't know if it is possible to link to a wiki document with just the > pageid, but it is possible to to get the url for the referring pageid via api: > http://en.wikipedia.org/w/api.php?action=query&prop=info&pageids=27697087&inprop=url > It is possible to get the metadata of a document using the pages id (instead > of title) directly: > Titel -> > http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API&rvprop=timestamp|user|comment|content > PageID -> > http://en.wikipedia.org/w/api.php?action=query&prop=revisions&pageids=27697087&rvprop=timestamp|user|comment|content > - There needs to be some notion of an overall list of pages: >- http://www.mediawiki.org/wiki/API:Allpages >- Example: > http://en.wikipedia.org/w/api.php?action=query&list=allpages&apfrom=Kre&aplimit=5 > - Metadata information (author and pub date) also needs to be separated out > in some way: >- http://www.mediawiki.org/wiki/API:Properties#Revisions:_Example >- Example: > http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API|Main%20Page&rvprop=timestamp|user|comment|content -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-100) DB lock timeout, and/or indefinite or excessive database activity
[
https://issues.apache.org/jira/browse/CONNECTORS-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128752#comment-13128752
]
Karl Wright edited comment on CONNECTORS-100 at 10/17/11 9:53 AM:
--
Comments in the Derby ticket indicate that this is a known problem with query
optimization in Derby, and thus they regard it as an enhancement to fix it.
We're therefore going to need to try and rejigger the queries in question to
use "IN" clauses, which Derby knows how to use, and hopefully not break any of
the other databases. PostgreSQL has certainly improved in this regard since
7.4, and HSQLDB seems pretty robust about handling clause transformations, so
all that I really need is a good test case to be able to verify that I haven't
broken anything.
Alternatively, I can try to come up with an abstraction which would allow more
flexibility in SQL generation between databases.
was (Author: kwri...@metacarta.com):
Comments in the Derby ticket indicate that this is a known problem with
query optimization in Derby, and thus they regard it as an enhancement to fix
it. We're therefore going to need to try and rejigger the queries in question
to use "IN" clauses, which Derby knows how to use, and hopefully not break any
of the other databases. PostgreSQL has certainly improved in this regard since
7.4, and HSQLDB seems pretty robust about handling clause transformations, so
all that I really need is a good test case to be able to verify that I haven't
broken anything.
> DB lock timeout, and/or indefinite or excessive database activity
> -
>
> Key: CONNECTORS-100
> URL: https://issues.apache.org/jira/browse/CONNECTORS-100
> Project: ManifoldCF
> Issue Type: Bug
> Components: Framework core
>Affects Versions: ManifoldCF 0.1, ManifoldCF 0.2, ManifoldCF 0.3,
> ManifoldCF 0.4
> Environment: Running unmodified dist/example from trunk/ using the
> default configuration.
>Reporter: Andrzej Bialecki
>Assignee: Karl Wright
> Fix For: ManifoldCF 0.4
>
>
> When a job is started and running (via crawler-ui) occasionally it's not
> possible to display a list of running jobs. The problem persists even after
> restarting ACF. The following exception is thrown in the console:
> {code}
> org.apache.acf.core.interfaces.ACFException: Database exception: Exception
> doing query: A lock could not be obtained within the time requested
> at
> org.apache.acf.core.database.Database.executeViaThread(Database.java:421)
> at
> org.apache.acf.core.database.Database.executeUncachedQuery(Database.java:465)
> at
> org.apache.acf.core.database.Database$QueryCacheExecutor.create(Database.java:1072)
> at
> org.apache.acf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.acf.core.database.Database.executeQuery(Database.java:167)
> at
> org.apache.acf.core.database.DBInterfaceDerby.performQuery(DBInterfaceDerby.java:727)
> at
> org.apache.acf.crawler.jobs.JobManager.makeJobStatus(JobManager.java:5611)
> at
> org.apache.acf.crawler.jobs.JobManager.getAllStatus(JobManager.java:5549)
> at
> org.apache.jsp.showjobstatus_jsp._jspService(showjobstatus_jsp.java:316)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
> at
> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpCon
[jira] [Issue Comment Edited] (CONNECTORS-279) Postgresql load test job delete document cleanup fails sometimes and leaves orphaned documents
[
https://issues.apache.org/jira/browse/CONNECTORS-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134787#comment-13134787
]
Karl Wright edited comment on CONNECTORS-279 at 10/25/11 6:52 AM:
--
Pertinent parts of stack dump:
{code}
"Document delete stuffer thread" daemon prio=6 tid=0x04947800 nid=0x119c in
Object.wait() [0x06b5f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.manifoldcf.core.lockmanager.LockObject.enterWriteLock(LockObject.java:111)
- locked <0x2c024058> (a
org.apache.manifoldcf.core.lockmanager.LockObject)
at
org.apache.manifoldcf.core.lockmanager.LockManager.enterWriteCriticalSection(LockManager.java:1459)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.noteModifications(DBInterfacePostgreSQL.java:1322)
at
org.apache.manifoldcf.core.database.BaseTable.noteModifications(BaseTable.java:299)
at
org.apache.manifoldcf.crawler.jobs.JobQueue.setDeletingStatus(JobQueue.java:718)
at
org.apache.manifoldcf.crawler.jobs.JobManager.getNextDeletableDocuments(JobManager.java:1227)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteStufferThread.run(DocumentDeleteStufferThread.java:105)
{code}
The thread that is probably holding the lock is:
{code}
"Document delete thread '0'" daemon prio=6 tid=0x04966c00 nid=0x13dc in
Object.wait() [0x06baf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1143)
- locked <0x2c021d08> (a
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
at java.lang.Thread.join(Thread.java:1196)
at
org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:453)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:505)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1152)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:639)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.reindexTableInternal(DBInterfacePostgreSQL.java:1284)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.noteModifications(DBInterfacePostgreSQL.java:1356)
at
org.apache.manifoldcf.core.database.BaseTable.noteModifications(BaseTable.java:299)
at
org.apache.manifoldcf.crawler.jobs.JobQueue.deleteIngestedDocumentIdentifiers(JobQueue.java:565)
at
org.apache.manifoldcf.crawler.jobs.JobManager.deleteIngestedDocumentIdentifiers(JobManager.java:789)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:176)
{code}
The query thread is:
{code}
"Thread-4367355" daemon prio=6 tid=0x04920c00 nid=0x1e88 runnable [0x04bef000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at
org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135)
at
org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104)
at
org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:255)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1165)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:191)
- locked <0x2c023e10> (a org.postgresql.core.v3.QueryExecutorImpl)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:452)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:337)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:329)
at
org.apache.manifoldcf.core.database.Database.execute(Database.java:566)
at
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:421)
{code}
So this looks like a case where the PostgreSQL JDBC driver is off chatting with
Postgres for the purposes of doing a reindex, but Postgres never responds back.
Given that I have no doubt many reindexes have taken place, I wonder why this
one is special?
was (Author: kwri...@metacarta.com):
Stack dump:
"Document delete stuffer thread" daemon prio=6 ti
[jira] [Issue Comment Edited] (CONNECTORS-287) An Alfresco connector would be helpful
[
https://issues.apache.org/jira/browse/CONNECTORS-287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158868#comment-13158868
]
Karl Wright edited comment on CONNECTORS-287 at 11/28/11 10:33 PM:
---
I built it and tried to run, w/o a repository to connect to, and got this:
{code}
org.alfresco.webservice.util.WebServiceException: Error creating authentication
service: java.net.MalformedURLException: no protocol: balh/AuthenticationService
at
org.alfresco.webservice.util.WebServiceFactory.getAuthenticationService(WebServiceFactory.java:167)
at
org.alfresco.webservice.util.AuthenticationUtils.startSession(AuthenticationUtils.java:73)
at
org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryConnector.getSession(AlfrescoRepositoryConnector.java:147)
at
org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryConnector.checkConnection(AlfrescoRepositoryConnector.java:179)
at
org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryConnector.check(AlfrescoRepositoryConnector.java:115)
at
org.apache.jsp.viewconnection_jsp._jspService(viewconnection_jsp.java:326)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at
org.apache.jasper.runtime.PageContextImpl.doForward(PageContextImpl.java:706)
at
org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:677)
...
{code|
This caused a blank screen to come back from the view connection page.
So I recommend the following:
(1) Since it seems like you need to enter a URL for the "endpoint" field, you
might want to consider requesting a protocol, server, port, and path instead of
a free-form URL. This will guarantee that the URL you construct will be valid.
(2) You should be catching any exceptions thrown by the API methods. I'm not
quite sure how you are getting something thrown that is not being caught,
unless the exception thrown is derived from RuntimeException or Error. In any
case you will need to figure out how to correctly catch and handle whatever
exceptions can be thrown by the API, and convert or wrap these in ManifoldCF
exceptions.
was (Author: kwri...@metacarta.com):
I built it and tried to run, w/o a repository to connect to, and got this:
org.alfresco.webservice.util.WebServiceException: Error creating authentication
service: java.net.MalformedURLException: no protocol: balh/AuthenticationService
at org.alfresco.webservice.util.WebServiceFactory.getAuthenticationServi
ce(WebServiceFactory.java:167)
at org.alfresco.webservice.util.AuthenticationUtils.startSession(Authent
icationUtils.java:73)
at org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryC
onnector.getSession(AlfrescoRepositoryConnector.java:147)
at org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryC
onnector.checkConnection(AlfrescoRepositoryConnector.java:179)
at org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryC
onnector.check(AlfrescoRepositoryConnector.java:115)
at org.apache.jsp.viewconnection_jsp._jspService(viewconnection_jsp.java
:326)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
.java:377)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3
13)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511
)
at org.mortbay.jetty.servlet.ServletHandler.handle(Servlet
[jira] [Issue Comment Edited] (CONNECTORS-287) An Alfresco connector would be helpful
[
https://issues.apache.org/jira/browse/CONNECTORS-287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158868#comment-13158868
]
Karl Wright edited comment on CONNECTORS-287 at 11/28/11 10:34 PM:
---
I built it and tried to run, w/o a repository to connect to, and got this:
{code}
org.alfresco.webservice.util.WebServiceException: Error creating authentication
service: java.net.MalformedURLException: no protocol: balh/AuthenticationService
at
org.alfresco.webservice.util.WebServiceFactory.getAuthenticationService(WebServiceFactory.java:167)
at
org.alfresco.webservice.util.AuthenticationUtils.startSession(AuthenticationUtils.java:73)
at
org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryConnector.getSession(AlfrescoRepositoryConnector.java:147)
at
org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryConnector.checkConnection(AlfrescoRepositoryConnector.java:179)
at
org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryConnector.check(AlfrescoRepositoryConnector.java:115)
at
org.apache.jsp.viewconnection_jsp._jspService(viewconnection_jsp.java:326)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at
org.apache.jasper.runtime.PageContextImpl.doForward(PageContextImpl.java:706)
at
org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:677)
...
{code}
This caused a blank screen to come back from the view connection page.
So I recommend the following:
(1) Since it seems like you need to enter a URL for the "endpoint" field, you
might want to consider requesting a protocol, server, port, and path instead of
a free-form URL. This will guarantee that the URL you construct will be valid.
(2) You should be catching any exceptions thrown by the API methods. I'm not
quite sure how you are getting something thrown that is not being caught,
unless the exception thrown is derived from RuntimeException or Error. In any
case you will need to figure out how to correctly catch and handle whatever
exceptions can be thrown by the API, and convert or wrap these in ManifoldCF
exceptions.
was (Author: kwri...@metacarta.com):
I built it and tried to run, w/o a repository to connect to, and got this:
{code}
org.alfresco.webservice.util.WebServiceException: Error creating authentication
service: java.net.MalformedURLException: no protocol: balh/AuthenticationService
at
org.alfresco.webservice.util.WebServiceFactory.getAuthenticationService(WebServiceFactory.java:167)
at
org.alfresco.webservice.util.AuthenticationUtils.startSession(AuthenticationUtils.java:73)
at
org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryConnector.getSession(AlfrescoRepositoryConnector.java:147)
at
org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryConnector.checkConnection(AlfrescoRepositoryConnector.java:179)
at
org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryConnector.check(AlfrescoRepositoryConnector.java:115)
at
org.apache.jsp.viewconnection_jsp._jspService(viewconnection_jsp.java:326)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler.handl
[jira] [Issue Comment Edited] (CONNECTORS-287) An Alfresco connector would be helpful
[
https://issues.apache.org/jira/browse/CONNECTORS-287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158868#comment-13158868
]
Karl Wright edited comment on CONNECTORS-287 at 11/28/11 10:35 PM:
---
I built it and tried to run, w/o a repository to connect to, and got this:
{code}
org.alfresco.webservice.util.WebServiceException: Error creating authentication
service: java.net.MalformedURLException: no protocol: balh/AuthenticationService
at
org.alfresco.webservice.util.WebServiceFactory.getAuthenticationService(WebServiceFactory.java:167)
at
org.alfresco.webservice.util.AuthenticationUtils.startSession(AuthenticationUtils.java:73)
at
org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryConnector.getSession(AlfrescoRepositoryConnector.java:147)
at
org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryConnector.checkConnection(AlfrescoRepositoryConnector.java:179)
at
org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryConnector.check(AlfrescoRepositoryConnector.java:115)
at
org.apache.jsp.viewconnection_jsp._jspService(viewconnection_jsp.java:326)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at
org.apache.jasper.runtime.PageContextImpl.doForward(PageContextImpl.java:706)
at
org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:677)
...
{code}
This caused a blank screen to come back from the view connection page.
So I recommend the following:
(1) Since it seems like you need to enter a URL for the "endpoint" field, you
might want to consider requesting a protocol, server, port, and path instead of
a free-form URL. Pick the protocol from a drop-down, and use javascript to
check that the server has no "/" characters and the port is either blank or a
number. This will guarantee that the URL you construct will be valid.
(2) You should be catching any exceptions thrown by the API methods. I'm not
quite sure how you are getting something thrown that is not being caught,
unless the exception thrown is derived from RuntimeException or Error. In any
case you will need to figure out how to correctly catch and handle whatever
exceptions can be thrown by the API, and convert or wrap these in ManifoldCF
exceptions.
was (Author: kwri...@metacarta.com):
I built it and tried to run, w/o a repository to connect to, and got this:
{code}
org.alfresco.webservice.util.WebServiceException: Error creating authentication
service: java.net.MalformedURLException: no protocol: balh/AuthenticationService
at
org.alfresco.webservice.util.WebServiceFactory.getAuthenticationService(WebServiceFactory.java:167)
at
org.alfresco.webservice.util.AuthenticationUtils.startSession(AuthenticationUtils.java:73)
at
org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryConnector.getSession(AlfrescoRepositoryConnector.java:147)
at
org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryConnector.checkConnection(AlfrescoRepositoryConnector.java:179)
at
org.apache.manifoldcf.crawler.connectors.alfresco.AlfrescoRepositoryConnector.check(AlfrescoRepositoryConnector.java:115)
at
org.apache.jsp.viewconnection_jsp._jspService(viewconnection_jsp.java:326)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java
[jira] [Issue Comment Edited] (CONNECTORS-308) "ant test" fails due to Alfresco test problem
[
https://issues.apache.org/jira/browse/CONNECTORS-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164435#comment-13164435
]
Karl Wright edited comment on CONNECTORS-308 at 12/7/11 2:48 PM:
-
Some research on the alfresco.war 3.4.d community edition: It is not
downloadable without considerable work. It's embedded in a zip file that can
only be downloaded from a server protected by basic auth, whose credentials are
supplied in Javascript in encoded form. It's under web-server/webapps in this
zip.
I'd therefore recommend that we put the onus on the user to supply this war
file in order for the integration tests to run. Any objection to this approach?
I'm not really sure what the Maven implications would be either, but clearly
the current approach of building the war ourselves using Alfresco materials is
certainly a concern. The license for the war itself is GPL, so we cannot just
check it into svn and be done with it either.
was (Author: kwri...@metacarta.com):
Some research on the alfresco.war 3.4.d community edition: It is not
downloadable without considerable work. It's embedded in a zip file that can
only be downloaded from a server protected by basic auth, whose credentials are
supplied in Javascript in encoded form. It's under web-server/webapps in this
zip.
I'd therefore recommend that we put the onus on the user to supply this war
file in order for the integration tests to run. Any objection to this approach?
I'm not really sure what the Maven implications would be either, but clearly
the current approach of building the war ourselves using Alfresco materials is
certainly a concern.
> "ant test" fails due to Alfresco test problem
> -
>
> Key: CONNECTORS-308
> URL: https://issues.apache.org/jira/browse/CONNECTORS-308
> Project: ManifoldCF
> Issue Type: Bug
> Components: Alfresco connector
>Affects Versions: ManifoldCF 0.4
> Environment: Windows (although this shouldn't matter)
>Reporter: Karl Wright
>Assignee: Piergiorgio Lucidi
> Fix For: ManifoldCF 0.4
>
>
> I get this error when I do "ant test":
> {code}
> ...
> setup-alfresco-connector-tests:
> [copy] Copying 6 files to C:\wip\mcf\trunk\connectors\alfresco\lib
> BUILD FAILED
> C:\wip\mcf\trunk\build.xml:316: The following error occurred while executing
> this line:
> C:\wip\mcf\trunk\tests\alfresco-war\build.xml:25: The following error
> occurred while executing this line:
> C:\wip\mcf\trunk\tests\alfresco-war\tools\ant\tasks\tomcat-tasks.xml:30:
> C:\wip\mcf\trunk\${appserver.dir}\server\lib does not exist.
> Total time: 18 seconds
> C:\wip\mcf\trunk>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-313) An example multi-process properties.xml delivered to the "dist" folder would be very helpful
[ https://issues.apache.org/jira/browse/CONNECTORS-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167421#comment-13167421 ] Karl Wright edited comment on CONNECTORS-313 at 12/12/11 9:17 AM: -- Hi Abe-san, connectors.xml is read by the QuickStart java startup class only at this time. You have two choices: (1) Modify the main build.xml to dynamically build your register.sh script, in a manner similar to the way it dynamically builds connectors.xml; (2) Create a new command class in framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler, which parses a connectors.xml and performs the corresponding modification (and nothing else), and then call that command class from your script. You may be able to move the code for this from framework/jettyrunner, and change the jettyrunner code so it is not duplicated. I'm not sure which is better. It depends on how people typically deploy the multiprocess version. was (Author: kwri...@metacarta.com): Hi Abe-san, connectors.xml is read by the QuickStart java startup class only at this time. You have two choices: (1) Modify the main build.xml to dynamically build your register.sh script, in a manner similar to the way it dynamically builds connectors.xml; (2) Create a new command class in framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler, which parses a connectors.xml and performs the corresponding modification (and nothing else). You may be able to move the code for this from framework/jettyrunner, and change the jettyrunner code so it is not duplicated. I'm not sure which is better. It depends on how people typically deploy the multiprocess version. > An example multi-process properties.xml delivered to the "dist" folder would > be very helpful > > > Key: CONNECTORS-313 > URL: https://issues.apache.org/jira/browse/CONNECTORS-313 > Project: ManifoldCF > Issue Type: Improvement > Components: Build >Affects Versions: ManifoldCF 0.4 >Reporter: Shinichiro Abe >Assignee: Shinichiro Abe >Priority: Minor > Fix For: ManifoldCF next > > Attachments: CONNECTOR-313.patch, CONNECTOR-313.patch, > register-draft.sh > > > The multiprocess setup does not have an example properties.xml file. We > should deliver one, in the right place so that all the scripts find it (the > "dist" directory). It would also be helpful to deliver into this directory > scripts for: > - Registering all the connectors that were built > - Starting the agents process -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-313) An example multi-process properties.xml delivered to the "dist" folder would be very helpful
[ https://issues.apache.org/jira/browse/CONNECTORS-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169181#comment-13169181 ] Karl Wright edited comment on CONNECTORS-313 at 12/14/11 8:36 AM: -- Do you think we should also have a script delivered in dist/multi that does all of the setup, such as setting MCF_HOME and calling all the right commands to initialize the database? For MCF_HOME, it could check whether './properties.xml' existed, and if it did set MCF_HOME to the value of '.', for instance. It should also optionally accept the database superuser name and superuser password (just like the DBCreate command), and perform the following command steps: - DBCreate - Initialize - RegisterAgent - RegisterAll The script can be located in: framework/example-multiprocess. What do you think? By the way, I also noticed that one of the Register commands you checked in is misspelled: Regsiter(?) was (Author: kwri...@metacarta.com): Do you think we should also have a script delivered in dist/multi that does all of the setup, such as setting MCF_HOME and calling all the right commands to initialize the database? > An example multi-process properties.xml delivered to the "dist" folder would > be very helpful > > > Key: CONNECTORS-313 > URL: https://issues.apache.org/jira/browse/CONNECTORS-313 > Project: ManifoldCF > Issue Type: Improvement > Components: Build >Affects Versions: ManifoldCF 0.4 >Reporter: Shinichiro Abe >Assignee: Shinichiro Abe >Priority: Minor > Fix For: ManifoldCF next > > Attachments: CONNECTOR-313.patch, CONNECTOR-313.patch, > register-draft.sh > > > The multiprocess setup does not have an example properties.xml file. We > should deliver one, in the right place so that all the scripts find it (the > "dist" directory). It would also be helpful to deliver into this directory > scripts for: > - Registering all the connectors that were built > - Starting the agents process -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-313) An example multi-process properties.xml delivered to the "dist" folder would be very helpful
[ https://issues.apache.org/jira/browse/CONNECTORS-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169916#comment-13169916 ] Karl Wright edited comment on CONNECTORS-313 at 12/15/11 2:05 AM: -- I think I see the problem. The class extends TransactionalCrawlerInitializationCommand, which presumes that the database instance already exists. I don't think you'll be able to extend that class if you are attempting database creation. I don't think it is necessary to do everything in one class anyway. The only new command functionality is the part that reads connectors.xml and registers the connectors. If you create a command that does just that, and a script that calls DBCreate, Initialize, RegisterAgent, and that, I think that would be fine. was (Author: kwri...@metacarta.com): I think I see the problem. The class extends TransactionalCrawlerInitializationCommand, which presumes that the database instance already exists. I don't think you'll be able to extend that class. I don't think it is necessary to do everything in one class anyway. The only new command functionality is the part that reads connectors.xml and registers the connectors. If you create a command that does just that, and a script that calls DBCreate, Initialize, RegisterAgent, and that, I think that would be fine. > An example multi-process properties.xml delivered to the "dist" folder would > be very helpful > > > Key: CONNECTORS-313 > URL: https://issues.apache.org/jira/browse/CONNECTORS-313 > Project: ManifoldCF > Issue Type: Improvement > Components: Build >Affects Versions: ManifoldCF 0.4 >Reporter: Shinichiro Abe >Assignee: Shinichiro Abe >Priority: Minor > Fix For: ManifoldCF 0.4 > > Attachments: CONNECTOR-313.patch, CONNECTOR-313.patch, > register-draft.sh > > > The multiprocess setup does not have an example properties.xml file. We > should deliver one, in the right place so that all the scripts find it (the > "dist" directory). It would also be helpful to deliver into this directory > scripts for: > - Registering all the connectors that were built > - Starting the agents process -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-313) An example multi-process properties.xml delivered to the "dist" folder would be very helpful
[ https://issues.apache.org/jira/browse/CONNECTORS-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169916#comment-13169916 ] Karl Wright edited comment on CONNECTORS-313 at 12/15/11 2:06 AM: -- I think I see the problem. The class extends TransactionalCrawlerInitializationCommand, which presumes that the database instance already exists and puts everything in a single transaction. I don't think you'll be able to extend that class if you are attempting database creation. I don't think it is necessary to do everything in one class anyway. The only new command functionality is the part that reads connectors.xml and registers the connectors. If you create a command that does just that, and a script that calls DBCreate, Initialize, RegisterAgent, and that, I think that would be fine. was (Author: kwri...@metacarta.com): I think I see the problem. The class extends TransactionalCrawlerInitializationCommand, which presumes that the database instance already exists. I don't think you'll be able to extend that class if you are attempting database creation. I don't think it is necessary to do everything in one class anyway. The only new command functionality is the part that reads connectors.xml and registers the connectors. If you create a command that does just that, and a script that calls DBCreate, Initialize, RegisterAgent, and that, I think that would be fine. > An example multi-process properties.xml delivered to the "dist" folder would > be very helpful > > > Key: CONNECTORS-313 > URL: https://issues.apache.org/jira/browse/CONNECTORS-313 > Project: ManifoldCF > Issue Type: Improvement > Components: Build >Affects Versions: ManifoldCF 0.4 >Reporter: Shinichiro Abe >Assignee: Shinichiro Abe >Priority: Minor > Fix For: ManifoldCF 0.4 > > Attachments: CONNECTOR-313.patch, CONNECTOR-313.patch, > register-draft.sh > > > The multiprocess setup does not have an example properties.xml file. We > should deliver one, in the right place so that all the scripts find it (the > "dist" directory). It would also be helpful to deliver into this directory > scripts for: > - Registering all the connectors that were built > - Starting the agents process -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-313) An example multi-process properties.xml delivered to the "dist" folder would be very helpful
[ https://issues.apache.org/jira/browse/CONNECTORS-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169950#comment-13169950 ] Karl Wright edited comment on CONNECTORS-313 at 12/15/11 3:12 AM: -- Oh, noticed one other thing. There's no un-registration of connectors. This is necessary because otherwise it is impossible to remove a connector using just RegisterAll. If you look at what the single process example does, it first deregisters all connectors before registering the new ones, for this reason. You may think that this is never going to happen with RegisterAll, but actually RegisterAll will be called not just for fresh installs but also for upgrades. was (Author: kwri...@metacarta.com): Oh, noticed one other thing. There's no un-registration of connectors. This is necessary because otherwise it is impossible to remove a connector using just RegisterAll. If you look at what the single process example does, it first deregisters all connectors before registering the new ones, for this reason. > An example multi-process properties.xml delivered to the "dist" folder would > be very helpful > > > Key: CONNECTORS-313 > URL: https://issues.apache.org/jira/browse/CONNECTORS-313 > Project: ManifoldCF > Issue Type: Improvement > Components: Build >Affects Versions: ManifoldCF 0.4 >Reporter: Shinichiro Abe >Assignee: Shinichiro Abe >Priority: Minor > Fix For: ManifoldCF 0.4 > > Attachments: CONNECTOR-313.patch, CONNECTOR-313.patch, > register-draft.sh > > > The multiprocess setup does not have an example properties.xml file. We > should deliver one, in the right place so that all the scripts find it (the > "dist" directory). It would also be helpful to deliver into this directory > scripts for: > - Registering all the connectors that were built > - Starting the agents process -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-323) *.sh in multiprocess-example doesn't work
[ https://issues.apache.org/jira/browse/CONNECTORS-323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170683#comment-13170683 ] Karl Wright edited comment on CONNECTORS-323 at 12/16/11 1:58 AM: -- Looks good! Please go ahead and commit to trunk. I'll pull it up to the release branch. was (Author: kwri...@metacarta.com): Looks good! Please go ahead and commit. > *.sh in multiprocess-example doesn't work > - > > Key: CONNECTORS-323 > URL: https://issues.apache.org/jira/browse/CONNECTORS-323 > Project: ManifoldCF > Issue Type: Bug > Components: Framework agents process >Affects Versions: ManifoldCF 0.4 >Reporter: Shinichiro Abe >Priority: Minor > Fix For: ManifoldCF 0.4 > > Attachments: CONNECTORS-323.patch > > > *.sh in multiprocess-example doesn't work.Here is a patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-330) HSQLDB external database tests fail
[
https://issues.apache.org/jira/browse/CONNECTORS-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171527#comment-13171527
]
Karl Wright edited comment on CONNECTORS-330 at 12/17/11 12:12 PM:
---
The story points back to HSQLDB again, unfortunately. Here's a trace:
{code}
[junit] Created HSQLDB instance
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB@113beb5 with user
manifoldcf which is meant to work with schema dbname
[junit] Created HSQLDB instance
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB@747fa2 with user
manifoldcf which is meant to work with schema dbname
[junit] Created HSQLDB instance
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB@6782a9 with user sa which
is meant to work with schema administration
[junit] Created HSQLDB instance
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB@17200b4 with user
manifoldcf which is meant to work with schema dbname
[junit] Created HSQLDB instance
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB@18c3679 with user sa
which is meant to work with schema administration
[junit] Creating schema DBNAME using instance
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB@18c3679
[junit] Altering user "manifoldcf"
[junit] Alter User succeeded! ALTER USER "manifoldcf" SET INITIAL SCHEMA
DBNAME
[junit] Created HSQLDB instance
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB@289d2e with user
manifoldcf which is meant to work with schema dbname
[junit] Checking for table in schema DBNAME in instance
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB@289d2e
[junit] Creating table in schema DBNAME in instance
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB@289d2e
[junit] org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
exception: Exception doing query: user lacks privilege or object not found:
PUBLIC in statement [CREATE CACHED TABLE agents(classname VARCHAR(255) NOT NULL
PRIMARY KEY)]
[junit] at
org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:672)
...
{code}
was (Author: kwri...@metacarta.com):
The story points back to HSQLDB again, unfortunately. Here's an annotated
trace:
{code}
<>
[junit] Created HSQLDB instance
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB@18c3679 with user sa
which is meant to work with schema administration
[junit] Creating schema DBNAME using instance
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB@18c3679
<>
[junit] Altering user "manifoldcf"
[junit] Alter User succeeded! ALTER USER "manifoldcf" SET INITIAL SCHEMA
DBNAME
<>
[junit] Created HSQLDB instance
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB@289d2e with user
manifoldcf which is meant to work with schema dbname
[junit] Checking for table in schema DBNAME in instance
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB@289d2e
[junit] Creating table in schema DBNAME in instance
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB@289d2e
[junit] org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
exception: Exception doing query: user lacks privilege or object not found:
PUBLIC in statement [CREATE CACHED TABLE agents(classname VARCHAR(255) NOT NULL
PRIMARY KEY)]
[junit] at
org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:672)
[junit] at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:716)
[junit] at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1363)
[junit] at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
[junit] at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:179)
[junit] at
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB.performModification(DBInterfaceHSQLDB.java:752)
[junit] at
org.apache.manifoldcf.core.database.DBInterfaceHSQLDB.performCreate(DBInterfaceHSQLDB.java:364)
[junit] at
org.apache.manifoldcf.core.database.BaseTable.performCreate(BaseTable.java:111)
[junit] at
org.apache.manifoldcf.agents.agentmanager.AgentManager.install(AgentManager.java:72)
[junit] at
org.apache.manifoldcf.agents.system.ManifoldCF.installTables(ManifoldCF.java:73)
[junit] at
org.apache.manifoldcf.agents.tests.BaseHSQLDBext.localSetUp(BaseHSQLDBext.java:65)
[junit] at
org.apache.manifoldcf.crawler.tests.BaseHSQLDBext.localSetUp(BaseHSQLDBext.java:60)
[junit] at
org.apache.manifoldcf.crawler.tests.BaseHSQLDBext.setUp(BaseHSQLDBext.java:48)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit] at
sun.reflect.Delegat
[jira] [Issue Comment Edited] (CONNECTORS-314) Combined MySQL and i18n/Japanese contribution
[ https://issues.apache.org/jira/browse/CONNECTORS-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172193#comment-13172193 ] Karl Wright edited comment on CONNECTORS-314 at 12/19/11 11:14 AM: --- bq. Is it necessary to include Message.java in each connector or is it possible to just leave it as a common utility class? A common utility class is fine. bq. There's "information schema" in MySQL. Just didn't have time this time to use it because it'll require more work. I wanted to show that it was possible to use MySQL with ManifoldCF and do the revisions afterwards. Somebody else may want to pitch in now since it's clear that it possible. As long as that is the case, I can lift the code from the hsqldb driver for the missing DBInterfaceMySQL methods. I will do that and check everything in when I've also coded the conditional download of the mysql driver in the build process. was (Author: kwri...@metacarta.com): bq. Is it necessary to include Message.java in each connector or is it possible to just leave it as a common utility class? A common utility class is fine. I'd move it to org.apache.manifoldcf.uicore.i18n though, I think, in the project framework/uicore. bq. There's "information schema" in MySQL. Just didn't have time this time to use it because it'll require more work. I wanted to show that it was possible to use MySQL with ManifoldCF and do the revisions afterwards. Somebody else may want to pitch in now since it's clear that it possible. As long as that is the case, I can lift the code from the hsqldb driver for the missing DBInterfaceMySQL methods. I will do that and check everything in when I've also coded the conditional download of the mysql driver in the build process. > Combined MySQL and i18n/Japanese contribution > - > > Key: CONNECTORS-314 > URL: https://issues.apache.org/jira/browse/CONNECTORS-314 > Project: ManifoldCF > Issue Type: Improvement > Components: Framework agents process, Framework core, Framework > crawler agent >Affects Versions: ManifoldCF 0.5 >Reporter: Karl Wright >Assignee: Karl Wright > Labels: I18N, mysql > Fix For: ManifoldCF 0.5 > > Attachments: CONNECTORS-314.patch, CONNECTORS-314.patch, > connectors.patch, framework.patch > > > Hitoshi Ozawa wishes to contribute i18n support, a Japanese localization, and > MySQL database support, all in one patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-314) Combined MySQL and i18n/Japanese contribution
[ https://issues.apache.org/jira/browse/CONNECTORS-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175232#comment-13175232 ] Karl Wright edited comment on CONNECTORS-314 at 12/23/11 2:13 AM: -- Hi Koji - I was wondering if maybe the .utf8 files are in fact the source files for the _ja.properties files, in which case we should probably keep the .utf8 files and generate the properties files dynamically. In fact I have confirmed that's how it works... so now I'm going to make appropriate rearrangements in the CONNECTORS-335 branch. was (Author: kwri...@metacarta.com): Hi Koji - the only thing I'm wondering is if maybe the .utf8 files are in fact the source files for the _ja.properties files, in which case we should probably keep the .utf8 files and generate the properties files dynamically. It's hard for me to tell what the intent was here. > Combined MySQL and i18n/Japanese contribution > - > > Key: CONNECTORS-314 > URL: https://issues.apache.org/jira/browse/CONNECTORS-314 > Project: ManifoldCF > Issue Type: Improvement > Components: Framework agents process, Framework core, Framework > crawler agent >Affects Versions: ManifoldCF 0.5 >Reporter: Karl Wright >Assignee: Karl Wright > Labels: I18N, mysql > Fix For: ManifoldCF 0.5 > > Attachments: CONNECTORS-314-doc20111220.patch, CONNECTORS-314.patch, > CONNECTORS-314.patch, connectors.patch, framework.patch > > > Hitoshi Ozawa wishes to contribute i18n support, a Japanese localization, and > MySQL database support, all in one patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-314) Combined MySQL and i18n/Japanese contribution
[ https://issues.apache.org/jira/browse/CONNECTORS-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176161#comment-13176161 ] Karl Wright edited comment on CONNECTORS-314 at 12/27/11 1:07 PM: -- bq. We can switch the properties file name to full language_country. I should have standardized it instead of using just language for some and full for others. What I'm going to try is to convert to <%=Message.getString()%> instead of for the JSP pages. This will allow everything to be consistent and run out of the same localization files. In addition, it should be possible to create localization escalation paths - e.g. if "ja_JP" is not there we look for "ja" and then "en_US" and then "en". That would minimize the maintenance task going forward. was (Author: kwri...@metacarta.com): bq. We can switch the properties file name to full language_country. I should have standardized it instead of using just language for some and full for others. What I'm going to try is to convert to <%=Message.getString()%> instead of for the JSP pages. This will allow everything to be consistent and run out of the same localization files. In addition, it should be possible to create localization escalation paths - e.g. if "ja_JP" is not there we look for "ja" and then "en_US" and then "en". That would minimize the maintenance task going forward. > Combined MySQL and i18n/Japanese contribution > - > > Key: CONNECTORS-314 > URL: https://issues.apache.org/jira/browse/CONNECTORS-314 > Project: ManifoldCF > Issue Type: Improvement > Components: Framework agents process, Framework core, Framework > crawler agent >Affects Versions: ManifoldCF 0.5 >Reporter: Karl Wright >Assignee: Karl Wright > Labels: I18N, mysql > Fix For: ManifoldCF 0.5 > > Attachments: CONNECTORS-314-20111225.patch, > CONNECTORS-314-doc20111220.patch, CONNECTORS-314.patch, CONNECTORS-314.patch, > connectors.patch, framework.patch > > > Hitoshi Ozawa wishes to contribute i18n support, a Japanese localization, and > MySQL database support, all in one patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-314) Combined MySQL and i18n/Japanese contribution
[ https://issues.apache.org/jira/browse/CONNECTORS-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13177047#comment-13177047 ] Karl Wright edited comment on CONNECTORS-314 at 12/29/11 7:50 AM: -- bq. Just want to make sure again. Should I be getting the source code from the trunk now? Where are fixes for the CONNECTORS-338 and CONNECTORS-340 being committed? All fixes have now been committed to trunk EXCEPT for the Japanese documentation, which is still going into branches/CONNECTORS-336. I'm also working on automated tests for the UI. Currently I'm testing just en_US, but I hope to add a set of ja_JP tests as well. As I go I am running into and fixing various minor UI bugs. That ticket is CONNECTORS-339, which is still open. was (Author: kwri...@metacarta.com): bq. Just want to make sure again. Should I be getting the source code from the trunk now? Where are fixes for the CONNECTORS-338 and CONNECTORS-340 being committed? All fixes have now been committed to trunk EXCEPT for the Japanese documentation, which is still going into branches/CONNECTORS-336. > Combined MySQL and i18n/Japanese contribution > - > > Key: CONNECTORS-314 > URL: https://issues.apache.org/jira/browse/CONNECTORS-314 > Project: ManifoldCF > Issue Type: Improvement > Components: Framework agents process, Framework core, Framework > crawler agent >Affects Versions: ManifoldCF 0.5 >Reporter: Karl Wright >Assignee: Karl Wright > Labels: I18N, mysql > Fix For: ManifoldCF 0.5 > > Attachments: CONNECTORS-314-20111225.patch, > CONNECTORS-314-doc20111220.patch, CONNECTORS-314.patch, CONNECTORS-314.patch, > connectors.patch, framework.patch > > > Hitoshi Ozawa wishes to contribute i18n support, a Japanese localization, and > MySQL database support, all in one patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-314) Combined MySQL and i18n/Japanese contribution
[ https://issues.apache.org/jira/browse/CONNECTORS-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13177047#comment-13177047 ] Karl Wright edited comment on CONNECTORS-314 at 12/29/11 7:57 AM: -- bq. Just want to make sure again. Should I be getting the source code from the trunk now? Where are fixes for the CONNECTORS-338 and CONNECTORS-340 being committed? All fixes have now been committed to trunk EXCEPT for the Japanese documentation, which is still going into branches/CONNECTORS-336. So yes, get your source code from trunk now. I do not anticipate that I will do further internationalization or localization-related changes unless I run into obvious problems when testing. So you are free to take your time and work on i18n and l10n changes without fear that I will change something out from underneath you. (I'm working on automated tests for the UI. Currently I'm testing just en_US, but I hope to add a set of ja_JP tests as well. As I go I am running into and fixing various minor UI bugs, but I should not be changing any infrastructure or organization as a result of these fixes. That ticket is CONNECTORS-339, which is still open.) was (Author: kwri...@metacarta.com): bq. Just want to make sure again. Should I be getting the source code from the trunk now? Where are fixes for the CONNECTORS-338 and CONNECTORS-340 being committed? All fixes have now been committed to trunk EXCEPT for the Japanese documentation, which is still going into branches/CONNECTORS-336. I'm also working on automated tests for the UI. Currently I'm testing just en_US, but I hope to add a set of ja_JP tests as well. As I go I am running into and fixing various minor UI bugs. That ticket is CONNECTORS-339, which is still open. > Combined MySQL and i18n/Japanese contribution > - > > Key: CONNECTORS-314 > URL: https://issues.apache.org/jira/browse/CONNECTORS-314 > Project: ManifoldCF > Issue Type: Improvement > Components: Framework agents process, Framework core, Framework > crawler agent >Affects Versions: ManifoldCF 0.5 >Reporter: Karl Wright >Assignee: Karl Wright > Labels: I18N, mysql > Fix For: ManifoldCF 0.5 > > Attachments: CONNECTORS-314-20111225.patch, > CONNECTORS-314-doc20111220.patch, CONNECTORS-314.patch, CONNECTORS-314.patch, > connectors.patch, framework.patch > > > Hitoshi Ozawa wishes to contribute i18n support, a Japanese localization, and > MySQL database support, all in one patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-341) override config for test cases
[
https://issues.apache.org/jira/browse/CONNECTORS-341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13177124#comment-13177124
]
Karl Wright edited comment on CONNECTORS-341 at 12/29/11 11:53 AM:
---
I had been toying with moving these embedded configuration files to
src/test/resource instead. It sounds like that would mess up your use case
entirely though.
I like the fact that "initialize()" can be separately overridden, but I'd like
perhaps to take it a step further and provide a base class method that is
specifically meant to be overridden such that the values written can be
extended rather than completely replaced, e.g.:
{code}
protected void populateConfigurationXML(StringBuilder sb)
{
...
}
{code}
... which is called from the base class implementation of initialize(). Also a
similar method for logging.ini. Do you understand what I mean, and if so do
you want to modify your patch accordingly?
was (Author: kwri...@metacarta.com):
I had been toying with moving these embedded configuration files to
src/test/resource instead. It sounds like that would mess up your use case
entirely though.
I like the fact that "initialize()" can be separately overridden, but I'd like
perhaps to take it a step further and provide a base class method that is
specifically meant to be overridden such that the values written can be
extended rather than completely replaced, e.g.:
protected void populateConfigurationXML(StringBuilder sb)
{
...
}
... which is called from the base class implementation of initialize(). Also a
similar method for logging.ini. Do you understand what I mean, and if so do
you want to modify your patch accordingly?
> override config for test cases
> --
>
> Key: CONNECTORS-341
> URL: https://issues.apache.org/jira/browse/CONNECTORS-341
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Tests
>Reporter: Martin Goldhahn
> Attachments: CONNECTORS-341.patch
>
>
> It is hard to write a test case for a repository connector using the existing
> *.Base classes that uses a different config file.
> The config files are actually written in
> org.apache.manifoldcf.core.tests.Base.localSetup().
> The localSetup method is called by all sub-classes in their overridden
> localSetup function. each sub-class adds something specific for the test
> case. If I wanted to put something different in the config files I had to
> override localSetup() and copy all content of the superclasses' localSetup to
> it. This can be avoided by moving the code that writes the config file in the
> initialize() method. It makes also more sense in that the creation of the
> config files and the provision of their content are now in the same function.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-336) If there's a Japanese localization, we should also have some Japanese documentation
[ https://issues.apache.org/jira/browse/CONNECTORS-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179461#comment-13179461 ] Karl Wright edited comment on CONNECTORS-336 at 1/4/12 1:18 PM: bq. I can't attach a zip here. It seems to hangup every time I try. How large is the zip? I believe the limit is 10mb (or maybe even 5mb), so if it's bigger than that you'll need to break it up into pieces. was (Author: kwri...@metacarta.com): bq. I can't attach a zip here. It seems to hangup every time I try. How large is the zip? > If there's a Japanese localization, we should also have some Japanese > documentation > --- > > Key: CONNECTORS-336 > URL: https://issues.apache.org/jira/browse/CONNECTORS-336 > Project: ManifoldCF > Issue Type: Improvement > Components: Documentation >Affects Versions: ManifoldCF 0.5 >Reporter: Karl Wright >Assignee: Karl Wright > Fix For: ManifoldCF 0.5 > > Attachments: CONNECTORS-336_20120104A.patch, > CONNECTORS-336_20120104B.patch > > > We should provide some Japanese documentation if we have a Japanese > localization. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-336) If there's a Japanese localization, we should also have some Japanese documentation
[ https://issues.apache.org/jira/browse/CONNECTORS-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179920#comment-13179920 ] Karl Wright edited comment on CONNECTORS-336 at 1/4/12 10:02 PM: - bq. The zip file I want to attach is about 8MB so it's under 10mb. Other people have uploaded zips to Jira successfully. Can you break it up into two roughly equal pieces and try that? 5Mb may be the limit. If that doesn't work just email the zip to me at daddy...@gmail.com, and I'll check it in. was (Author: kwri...@metacarta.com): bq. The zip file I want to attach is about 8MB so it's under 10mb. Other people have uploaded zips to Jira successfully. Can you break it up into two roughly equal pieces and try that? 5Mb may be the limit. > If there's a Japanese localization, we should also have some Japanese > documentation > --- > > Key: CONNECTORS-336 > URL: https://issues.apache.org/jira/browse/CONNECTORS-336 > Project: ManifoldCF > Issue Type: Improvement > Components: Documentation >Affects Versions: ManifoldCF 0.5 >Reporter: Karl Wright >Assignee: Karl Wright > Fix For: ManifoldCF 0.5 > > Attachments: CONNECTORS-336_20120104A.patch, > CONNECTORS-336_20120104B.patch > > > We should provide some Japanese documentation if we have a Japanese > localization. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-336) If there's a Japanese localization, we should also have some Japanese documentation
[ https://issues.apache.org/jira/browse/CONNECTORS-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180356#comment-13180356 ] Karl Wright edited comment on CONNECTORS-336 at 1/5/12 1:20 PM: The new patch fails on tab.xml and site.xml: patching file xdocs/tabs.xml Reversed (or previously applied) patch detected! Assume -R? [n] y patching file xdocs/site.xml Reversed (or previously applied) patch detected! Assume -R? [n] y Hunk #2 FAILED at 75. 1 out of 2 hunks FAILED -- saving rejects to file xdocs/site.xml.rej Are you sure you did an svn diff against the current contents of the branch? Try "svn update" before you do the "svn diff". was (Author: kwri...@metacarta.com): The new patch fails on tab.xml and site.xml: patching file xdocs/tabs.xml Reversed (or previously applied) patch detected! Assume -R? [n] y patching file xdocs/site.xml Reversed (or previously applied) patch detected! Assume -R? [n] y Hunk #2 FAILED at 75. 1 out of 2 hunks FAILED -- saving rejects to file xdocs/site.xml.rej Are you sure you did an svn diff against the current contents of the branch? > If there's a Japanese localization, we should also have some Japanese > documentation > --- > > Key: CONNECTORS-336 > URL: https://issues.apache.org/jira/browse/CONNECTORS-336 > Project: ManifoldCF > Issue Type: Improvement > Components: Documentation >Affects Versions: ManifoldCF 0.5 >Reporter: Karl Wright >Assignee: Karl Wright > Fix For: ManifoldCF 0.5 > > Attachments: CONNECTORS-336_20120104A.patch, > CONNECTORS-336_20120104B.patch, CONNECTORS-336_20120105.patch, ja_JP_A.zip, > ja_JP_B.zip, ja_JP_C.zip > > > We should provide some Japanese documentation if we have a Japanese > localization. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-286) Get ManifoldCF to run on top of a key/value store like Voldemort, for potential massive scalability improvements and speed gains
[ https://issues.apache.org/jira/browse/CONNECTORS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181155#comment-13181155 ] Karl Wright edited comment on CONNECTORS-286 at 1/6/12 6:23 AM: bq. Not sure what Wharthog is... See https://svn.apache.org/repos/asf/incubator/lcf/branches/CONNECTORS-286/warthog. "Warthog" is a potential future spinoff technology of ManifoldCF - if all this stuff actually works. ;-) bq. But I wanted to suggest you consider using HBase for underlying storage. Maybe. One of the apparent requirements of ManifoldCF is the ability run on top of practically anything. I don't think we'd be able to throw away PostgreSQL and MySQL and HSQLDB and Derby support for instance. So if Warthog is the actual API layer ManifoldCF uses then we'd need implementations of Warthog for as many backends as possible. Right now I've only got one going for a testing key-value store, but almost certainly the next step would be a SQL database. Then moving on from then HBase (or Hive/Pig) may also be possibilities. Still proving the concept however... was (Author: kwri...@metacarta.com): bq. Not sure what Wharthog is... See https://svn.apache.org/repos/asf/incubator/lcf/branches/CONNECTORS-286/warthog. "Warthog" is a potential future spinoff technology of ManifoldCF - if all this stuff actually works. ;-) bq. But I wanted to suggest you consider using HBase for underlying storage. Maybe. One of the apparent requirements of ManifoldCF is the ability run on top of practically anything. I don't think we'd be able to throw away PostgreSQL and MySQL and HSQLDB and Derby support for instance. So if Warthog is the actual API layer ManifoldCF uses then we'd need implementations of Warthog for as many backends as possible. Right now I've only going one for a testing key-value store, but almost certainly the next step would be a SQL database. Then moving on from then HBase (or Hive/Pig) may also be possibilities. Still proving the concept however... > Get ManifoldCF to run on top of a key/value store like Voldemort, for > potential massive scalability improvements and speed gains > > > Key: CONNECTORS-286 > URL: https://issues.apache.org/jira/browse/CONNECTORS-286 > Project: ManifoldCF > Issue Type: New Feature > Components: Framework core >Reporter: Karl Wright >Assignee: Karl Wright > Fix For: ManifoldCF next > > > ManifoldCF's reliance on a relational database limits its throughput and > scalability. I am now convinced it is possible to build all the structures > we need within a distributed key-value store like Voldemort, which has the > nice side effect of permitting massive scaling. I envision there will be > several layers to this project, some of which may have broader utility in the > open-source community at large: > (1) An atomic serialization layer, which adds serialization capabilities to > an non-transactional substrate; > (2) A transaction layer, which uses atomic serialization to build a notion of > light transactions; > (3) A table and index layer, which defines SQL-like concepts of tables and > btree indexes on top of the transaction layer, via a Java API; > (4) A generic "database abstraction" layer, which is capable of representing > both standard SQL databases as well as this NoSQL variant, so that ManifoldCF > can support both models. > This is obviously a major development task, and as such is not envisioned to > be completed by the next standard release. Work will indeed need to be done > in a branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-336) If there's a Japanese localization, we should also have some Japanese documentation
[ https://issues.apache.org/jira/browse/CONNECTORS-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181387#comment-13181387 ] Karl Wright edited comment on CONNECTORS-336 at 1/6/12 3:26 PM: bq. I may have messed up my svn on Ubuntu by copying files I've edited on my Windows7 Copying files shouldn't mess up svn, but copying workspaces/hidden files certainly could. If you're not sure it may be best to create a new workspace at this point. Once you do that, you should notice that there's no ja_JP/index*.xml file. Then you can add yours back and do an svn diff to get the next patch. was (Author: kwri...@metacarta.com): bq. I may have messed up my svn on Ubuntu by copying files I've edited on my Windows7 Copying files shouldn't mess up svn, but copying workspaces/hidden files certainly could. If you're not sure it may be best to create a new workspace at this point. > If there's a Japanese localization, we should also have some Japanese > documentation > --- > > Key: CONNECTORS-336 > URL: https://issues.apache.org/jira/browse/CONNECTORS-336 > Project: ManifoldCF > Issue Type: Improvement > Components: Documentation >Affects Versions: ManifoldCF 0.5 >Reporter: Karl Wright >Assignee: Karl Wright > Fix For: ManifoldCF 0.5 > > Attachments: CONNECTORS-336-20120106.patch, > CONNECTORS-336_20120104A.patch, CONNECTORS-336_20120104B.patch, > CONNECTORS-336_20120105.patch, ja_JP_A.zip, ja_JP_B.zip, ja_JP_C.zip > > > We should provide some Japanese documentation if we have a Japanese > localization. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-356) Active Directory connector should by fully I18N/L10N
[
https://issues.apache.org/jira/browse/CONNECTORS-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182492#comment-13182492
]
Karl Wright edited comment on CONNECTORS-356 at 1/9/12 1:52 PM:
I've looked at this patch. It embeds the Javascript etc. quotation marks right
in the translation. I believe this to be bad practice since the localization
is then based on the context in which it is used.
For example:
{code}
+"alert(" +
Messages.getString(locale,"ActiveDirectoryAuthority.EnterADomainControllerServerName")
+ ");\n"+
{code}
...combined with...
{code}
+ActiveDirectoryAuthority.EnterADomainControllerServerName=\"Enter a domain
controller server name\"
{code}
I think this should be:
{code}
+"alert(\"" +
Messages.getBodyJavascriptString(locale,"ActiveDirectoryAuthority.EnterADomainControllerServerName")
+ "\");\n"+
{code}
... and ...
{code}
+ActiveDirectoryAuthority.EnterADomainControllerServerName=Enter a domain
controller server name
{code}
Do you agree?
was (Author: kwri...@metacarta.com):
I've looked at this patch. It embeds the Javascript etc. quotation marks
right in the translation. I believe this to be bad practice since the
localization is then based on the context in which it is used.
For example:
{code}
+"alert(" +
Messages.getString(locale,"ActiveDirectoryAuthority.EnterADomainControllerServerName")
+ ");\n"+
{code}
...combined with...
{code}
+ActiveDirectoryAuthority.EnterADomainControllerServerName=\"Enter a domain
controller server name\"
{code}
I think this should be:
{code}
+"alert(\"" +
Messages.getJavascriptBodyString(locale,"ActiveDirectoryAuthority.EnterADomainControllerServerName")
+ "\");\n"+
{code}
... and ...
{code}
+ActiveDirectoryAuthority.EnterADomainControllerServerName=Enter a domain
controller server name
{code}
Do you agree?
> Active Directory connector should by fully I18N/L10N
>
>
> Key: CONNECTORS-356
> URL: https://issues.apache.org/jira/browse/CONNECTORS-356
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Active Directory authority
>Affects Versions: ManifoldCF 0.5
>Reporter: Hitoshi Ozawa
>Assignee: Karl Wright
>Priority: Minor
> Labels: I18N
> Fix For: ManifoldCF 0.5
>
> Attachments: CONNECTORS-356.patch
>
>
> All messages in Active Directory Connector should be extracted out to
> properties file.
> Also, should create Japanese L10N properties file for all messages.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-388) Upgrade the Crawler UI framework with Apache Velocity
[ https://issues.apache.org/jira/browse/CONNECTORS-388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188743#comment-13188743 ] Karl Wright edited comment on CONNECTORS-388 at 1/18/12 10:18 PM: -- I'm refactoring the root build.xml to build up the manifest classpath a line at a time, so we'll have to merge that when the time comes. Other than that, good work! The reason that VelocityEngine doesn't find the resources is because they can only be accessed by the current classloader, which is defined in ManifoldCF core as a ordered list of classloaders. You can get a new classloader based on the current list by using the method ManifoldCF.createResourceLoader(). But preferably, if you use the local connector's Messages.class.getClassLoader(), you'll get the class loader used to load the local connector's Messages class. This is typically the right classloader. That's why I created these classes, so there'd be an easy way to do it. was (Author: kwri...@metacarta.com): I'm refactoring the root build.xml to build up the manifest classpath a line at a time, so we'll have to merge that when the time comes. Other than that, good work! The reason that VelocityEngine doesn't find the resources is because they can only be accessed by the current classloader, which is defined in ManifoldCF core as a nested set of classloaders. But you can get the correct classloader using the method ManifoldCF.createResourceLoader(). > Upgrade the Crawler UI framework with Apache Velocity > - > > Key: CONNECTORS-388 > URL: https://issues.apache.org/jira/browse/CONNECTORS-388 > Project: ManifoldCF > Issue Type: Improvement > Components: Framework core >Affects Versions: ManifoldCF 0.3 >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi > Original Estimate: 72h > Remaining Estimate: 72h > > We have to try to upgrade the UI Crawler using Apache Velocity that a good > solution that could fit with Manifold implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-387) Japanese fonts are missing from the PDF files generated for the site
[ https://issues.apache.org/jira/browse/CONNECTORS-387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193606#comment-13193606 ] Karl Wright edited comment on CONNECTORS-387 at 1/26/12 7:23 AM: - Hitoshi, I don't see any sign that Apache Legal has ever ruled on the IPA license, but several clauses in there give me reason to question whether they would permit us to redistribute it. I'm happy to post the license and have them rule but it will take some time. What we'll also need to do is figure out a way to switch between fonts based on the path of the PDF being generated if we stick with IPA, because a clause in the license definitely forbids relabeling or modifying the IPA fonts, so we could not redistribute a merged font family. That's why I've been exploring the Dejavu route, since the Forrest page explicitly mentioned CJK support in that family. Too bad it isn't true. was (Author: kwri...@metacarta.com): Hitoshi, I don't see any sign that Apache Legal has ever ruled on the IPA license, but several clauses in there give me reason to question whether they would permit us to redistribute it. I'm happy to post the license and have them rule but it will take some time. If IPA is the only available choice I think we're going to have to make do without Japanese PDFs for the time being. What we'll also need to do is figure out a way to switch between fonts based on the path of the PDF being generated if we stick with IPA. That's why I've been exploring the Dejavu route, since the Forrest page explicitly mentioned CJK support in that family. > Japanese fonts are missing from the PDF files generated for the site > > > Key: CONNECTORS-387 > URL: https://issues.apache.org/jira/browse/CONNECTORS-387 > Project: ManifoldCF > Issue Type: Bug > Components: Documentation >Affects Versions: ManifoldCF 0.5 >Reporter: Karl Wright >Assignee: Karl Wright > Fix For: ManifoldCF 0.5 > > > Japanese fonts are missing from the PDFs generated for the site. > This page describes how to fix that: > http://forrest.apache.org/pluginDocs/plugins_0_90/org.apache.forrest.plugin.output.pdf/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-387) Japanese fonts are missing from the PDF files generated for the site
[
https://issues.apache.org/jira/browse/CONNECTORS-387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195452#comment-13195452
]
Karl Wright edited comment on CONNECTORS-387 at 1/28/12 8:17 AM:
-
I get the following when the PDFs are being built under the above setup:
{code}
[exec] WARN - Font 'IPAPGothic,normal,400' not found. Substituting with
'any,normal,400'.
[exec] WARN - Font 'IPAPGothic,normal,700' not found. Substituting with
'any,normal,700'.
[exec] WARN - Font 'IPAPGothic,italic,400' not found. Substituting with
'any,italic,400'.
{code}
It's not clear whether this is because none of the IPAP Gothic fonts are
properly registered, or because only these three fonts are needed. Looking
further, it doesn't look like any of the IPAP fonts are being properly
registered. This is when I put config.xml at:
src/documentation/resources/fonts/config.xml. Clearly that's not the right
place, or I've done something wrong in it.
I'm checking the whole thing in now, since it doesn't seem to break anything
worse than it is already broken.
was (Author: kwri...@metacarta.com):
I get the following when the PDFs are being built under the above setup:
{code}
[exec] WARN - Font 'IPAPGothic,normal,400' not found. Substituting with
'any,normal,400'.
[exec] WARN - Font 'IPAPGothic,normal,700' not found. Substituting with
'any,normal,700'.
[exec] WARN - Font 'IPAPGothic,italic,400' not found. Substituting with
'any,italic,400'.
{code}
It's not clear whether this is because none of the IPAP Gothic fonts are
properly registered, or because only these three fonts are needed. Looking
further...
> Japanese fonts are missing from the PDF files generated for the site
>
>
> Key: CONNECTORS-387
> URL: https://issues.apache.org/jira/browse/CONNECTORS-387
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentation
>Affects Versions: ManifoldCF 0.5
>Reporter: Karl Wright
>Assignee: Karl Wright
> Fix For: ManifoldCF 0.5
>
>
> Japanese fonts are missing from the PDFs generated for the site.
> This page describes how to fix that:
> http://forrest.apache.org/pluginDocs/plugins_0_90/org.apache.forrest.plugin.output.pdf/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-387) Japanese fonts are missing from the PDF files generated for the site
[
https://issues.apache.org/jira/browse/CONNECTORS-387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195452#comment-13195452
]
Karl Wright edited comment on CONNECTORS-387 at 1/28/12 8:23 AM:
-
I get the following when the PDFs are being built under the above setup:
{code}
[exec] WARN - Font 'IPAPGothic,normal,400' not found. Substituting with
'any,normal,400'.
[exec] WARN - Font 'IPAPGothic,normal,700' not found. Substituting with
'any,normal,700'.
[exec] WARN - Font 'IPAPGothic,italic,400' not found. Substituting with
'any,italic,400'.
{code}
It's not clear whether this is because none of the IPAP Gothic fonts are
properly registered, or because only these three fonts are needed. Looking
further, it doesn't look like any of the IPAP fonts are being properly
registered. This is where I put config.xml at:
src/documentation/resources/fonts/config.xml. Clearly that's not the right
place, or I've done something wrong in it.
I'm checking the whole thing in now, since it doesn't seem to break anything
worse than it is already broken.
was (Author: kwri...@metacarta.com):
I get the following when the PDFs are being built under the above setup:
{code}
[exec] WARN - Font 'IPAPGothic,normal,400' not found. Substituting with
'any,normal,400'.
[exec] WARN - Font 'IPAPGothic,normal,700' not found. Substituting with
'any,normal,700'.
[exec] WARN - Font 'IPAPGothic,italic,400' not found. Substituting with
'any,italic,400'.
{code}
It's not clear whether this is because none of the IPAP Gothic fonts are
properly registered, or because only these three fonts are needed. Looking
further, it doesn't look like any of the IPAP fonts are being properly
registered. This is when I put config.xml at:
src/documentation/resources/fonts/config.xml. Clearly that's not the right
place, or I've done something wrong in it.
I'm checking the whole thing in now, since it doesn't seem to break anything
worse than it is already broken.
> Japanese fonts are missing from the PDF files generated for the site
>
>
> Key: CONNECTORS-387
> URL: https://issues.apache.org/jira/browse/CONNECTORS-387
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentation
>Affects Versions: ManifoldCF 0.5
>Reporter: Karl Wright
>Assignee: Karl Wright
> Fix For: ManifoldCF 0.5
>
>
> Japanese fonts are missing from the PDFs generated for the site.
> This page describes how to fix that:
> http://forrest.apache.org/pluginDocs/plugins_0_90/org.apache.forrest.plugin.output.pdf/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-387) Japanese fonts are missing from the PDF files generated for the site
[ https://issues.apache.org/jira/browse/CONNECTORS-387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195860#comment-13195860 ] Karl Wright edited comment on CONNECTORS-387 at 1/29/12 10:19 PM: -- Upgrading to the trunk version of fop fixed the problem. There was also an issue with a duplicate copy of the 0.95 jar hanging around, so the 1.0 fop jar might well have worked if I'd noticed that before. r1237495. Next step is to look into whether output.xmap can be conditionalized to use different fonts for different parts of the tree. was (Author: kwri...@metacarta.com): Upgrading to the trunk version of fop fixed the problem. r1237495. Next step is to look into whether output.xmap can be conditionalized to use different fonts for different parts of the tree. > Japanese fonts are missing from the PDF files generated for the site > > > Key: CONNECTORS-387 > URL: https://issues.apache.org/jira/browse/CONNECTORS-387 > Project: ManifoldCF > Issue Type: Bug > Components: Documentation >Affects Versions: ManifoldCF 0.5 >Reporter: Karl Wright >Assignee: Karl Wright > Fix For: ManifoldCF 0.5 > > > Japanese fonts are missing from the PDFs generated for the site. > This page describes how to fix that: > http://forrest.apache.org/pluginDocs/plugins_0_90/org.apache.forrest.plugin.output.pdf/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-387) Japanese fonts are missing from the PDF files generated for the site
[ https://issues.apache.org/jira/browse/CONNECTORS-387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195994#comment-13195994 ] Karl Wright edited comment on CONNECTORS-387 at 1/30/12 8:43 AM: - Looks good, and the properly-generated PDFs are already live. I've rearranged the site code now to give more equal emphasis to english and japanese, and also allow for the crawler UI "help" like to take you to the right help page for your language, but I have not published this yet. The only downside here is that there's now need of a "launch" tab which does nothing more than tell you to click on the appropriate language tab. I'm hoping Google will pick this up OK. Hitoshi, could you synch up trunk and do "ant download-dependencies doc", and see that the site navigation for the site at dist/doc now makes sense in Japanese? The only outstanding work for this ticket now is trying to get different fonts used to generate PDFs for different parts of the site. was (Author: kwri...@metacarta.com): Looks good, and the properly-generated PDFs are already live. I've rearranged the site to give more equal emphasis to english and japanese, and also allow for the crawler UI "help" like to take you to the right help page for your language. The only downside here is that there's now need of a "launch" tab which does nothing more than tell you to click on the appropriate language tab. I'm hoping Google will pick this up OK. Hitoshi, could you synch up trunk and do "ant download-dependencies doc", and see that the site navigation for the site at dist/doc now makes sense in Japanese? The only outstanding work for this ticket now is trying to get different fonts used to generate PDFs for different parts of the site. > Japanese fonts are missing from the PDF files generated for the site > > > Key: CONNECTORS-387 > URL: https://issues.apache.org/jira/browse/CONNECTORS-387 > Project: ManifoldCF > Issue Type: Bug > Components: Documentation >Affects Versions: ManifoldCF 0.5 >Reporter: Karl Wright >Assignee: Karl Wright > Fix For: ManifoldCF 0.5 > > > Japanese fonts are missing from the PDFs generated for the site. > This page describes how to fix that: > http://forrest.apache.org/pluginDocs/plugins_0_90/org.apache.forrest.plugin.output.pdf/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-387) Japanese fonts are missing from the PDF files generated for the site
[ https://issues.apache.org/jira/browse/CONNECTORS-387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195994#comment-13195994 ] Karl Wright edited comment on CONNECTORS-387 at 1/30/12 10:26 AM: -- Looks good, and the properly-generated PDFs are already live. I've rearranged the site code now to give more equal emphasis to english and japanese, and also allow for the crawler UI "help" link to take you to the right help page for your language, but I have not published this yet. The only downside here is that there's now need of a "launch" tab which does nothing more than tell you to click on the appropriate language tab. I'm hoping Google will pick this up OK. Hitoshi, could you synch up trunk and do "ant download-dependencies doc", and see that the site navigation for the site at dist/doc now makes sense in Japanese? The only outstanding work for this ticket now is trying to get different fonts used to generate PDFs for different parts of the site. was (Author: kwri...@metacarta.com): Looks good, and the properly-generated PDFs are already live. I've rearranged the site code now to give more equal emphasis to english and japanese, and also allow for the crawler UI "help" like to take you to the right help page for your language, but I have not published this yet. The only downside here is that there's now need of a "launch" tab which does nothing more than tell you to click on the appropriate language tab. I'm hoping Google will pick this up OK. Hitoshi, could you synch up trunk and do "ant download-dependencies doc", and see that the site navigation for the site at dist/doc now makes sense in Japanese? The only outstanding work for this ticket now is trying to get different fonts used to generate PDFs for different parts of the site. > Japanese fonts are missing from the PDF files generated for the site > > > Key: CONNECTORS-387 > URL: https://issues.apache.org/jira/browse/CONNECTORS-387 > Project: ManifoldCF > Issue Type: Bug > Components: Documentation >Affects Versions: ManifoldCF 0.5 >Reporter: Karl Wright >Assignee: Karl Wright > Fix For: ManifoldCF 0.5 > > > Japanese fonts are missing from the PDFs generated for the site. > This page describes how to fix that: > http://forrest.apache.org/pluginDocs/plugins_0_90/org.apache.forrest.plugin.output.pdf/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-402) Separate the proprietary / non-proprietary deliverables for each connector
[ https://issues.apache.org/jira/browse/CONNECTORS-402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204068#comment-13204068 ] Karl Wright edited comment on CONNECTORS-402 at 2/8/12 11:06 PM: - I created a branch CONNECTORS-402 for this work. Problems still outstanding: (1) Since JDBC drivers must be built into the war files, they cannot be delivered into connector-lib. This means we have no ability to prebuild the JDBC connector to include oracle, mssql, sybase, or mysql support. We need a way of suppressing inclusion of these JDBC jars when building for release, even though we will likely have run the "ant download-dependencies" target in the release branch. (2) The tests all need to be run and checked, since the place where proprietary materials are placed has now become "lib-proprietary" across the board. (3) The how-to-build-and-deploy documentation needs significant changes. (4) The process components of the FileNet and Documentum connectors are where the proprietary jars wind up; we need a solution for these. (5) How to make commented-out connectors.xml entries when building for release, for those connectors that have proprietary components. was (Author: kwri...@metacarta.com): I created a branch CONNECTORS-402 for this work. Problems still outstanding: (1) Since JDBC drivers must be built into the war files, they cannot be delivered into connector-lib. This means we have no ability to prebuild the JDBC connector to include oracle, mssql, sybase, or mysql support. We need a way of suppressing inclusion of these JDBC jars when building for release, even though we will likely have run the "ant download-dependencies" target in the release branch. (2) The tests all need to be run and checked, since the place where proprietary materials are placed has now become "lib-proprietary" across the board. (3) The how-to-build-and-deploy documentation needs significant changes. (4) The process components of the FileNet and Documentum connectors are where the proprietary jars wind up; we need a solution for these. > Separate the proprietary / non-proprietary deliverables for each connector > -- > > Key: CONNECTORS-402 > URL: https://issues.apache.org/jira/browse/CONNECTORS-402 > Project: ManifoldCF > Issue Type: Improvement > Components: Build >Affects Versions: ManifoldCF 0.4 >Reporter: Piergiorgio Lucidi >Assignee: Karl Wright > Fix For: ManifoldCF 0.5 > > Original Estimate: 24h > Remaining Estimate: 24h > > As we have spoken today this ticket is dedicated to try to work on the build > process to separate proprietary and non-proprietary libraries in the release > package. > In this way, for example, we could have a standard way to release a specific > library for the connector (mcf-alfresco-connector) without release > proprietary libraries and allow users to add proprietary libraries in a > specific folder "connector-lib-proprietary" that could be added to the > ManifoldCF classloader. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-288) An ElasticSearch connector would be helpful
[
https://issues.apache.org/jira/browse/CONNECTORS-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209673#comment-13209673
]
Karl Wright edited comment on CONNECTORS-288 at 2/16/12 8:17 PM:
-
I modified the ant build to properly build and run the tests. Unfortunately
the compilation of the IT tests fails:
{code}
compile-tests:
[javac] C:\wip\mcf\CONNECTORS-288\tests\test-build.xml:102: warning: 'includ
eantruntime' was not set, defaulting to build.sysclasspath=last; set to false fo
r repeatable builds
[javac] Compiling 4 source files to C:\wip\mcf\CONNECTORS-288\tests\elastics
earch\build\test\classes
[javac] C:\wip\mcf\CONNECTORS-288\tests\elasticsearch\src\test\java\org\apac
he\manifoldcf\elasticsearch_tests\BaseDerby.java:22: package org.elasticsearch.n
ode does not exist
[javac] import org.elasticsearch.node.Node;
[javac] ^
[javac] C:\wip\mcf\CONNECTORS-288\tests\elasticsearch\src\test\java\org\apac
he\manifoldcf\elasticsearch_tests\BaseDerby.java:27: package org.elasticsearch.n
ode does not exist
[javac] import static org.elasticsearch.node.NodeBuilder.*;
[javac] ^
[javac] C:\wip\mcf\CONNECTORS-288\tests\elasticsearch\src\test\java\org\apac
he\manifoldcf\elasticsearch_tests\BaseDerby.java:37: cannot find symbol
[javac] symbol : class Node
[javac] location: class org.apache.manifoldcf.elasticsearch_tests.BaseDerby
[javac] protected Node node = null;
[javac] ^
[javac] C:\wip\mcf\CONNECTORS-288\tests\elasticsearch\src\test\java\org\apac
he\manifoldcf\elasticsearch_tests\BaseDerby.java:92: cannot find symbol
[javac] symbol : method nodeBuilder()
[javac] location: class org.apache.manifoldcf.elasticsearch_tests.BaseDerby
[javac] node = nodeBuilder().local(true).node();
[javac]^
[javac] 4 errors
{code}
was (Author: kwri...@metacarta.com):
I modified the ant build to properly build and run the tests.
Unfortunately the compilation of the IT tests fails.
> An ElasticSearch connector would be helpful
> ---
>
> Key: CONNECTORS-288
> URL: https://issues.apache.org/jira/browse/CONNECTORS-288
> Project: ManifoldCF
> Issue Type: New Feature
>Affects Versions: ManifoldCF 0.5
>Reporter: Piergiorgio Lucidi
>Assignee: Piergiorgio Lucidi
> Labels: elasticsearch
> Fix For: ManifoldCF next
>
> Attachments: manifold-elasticsearch-patch,
> manifold-elasticsearch-patch, manifold-elasticsearch-patch,
> manifold-elasticsearch-patch, manifold-elasticsearch-patch,
> manifold-elasticsearch-patch, manifoldcf-elasticsearch-project-patct
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> An ElasticSearch connector could be very useful to spread the use of
> ManifoldCF
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-288) An ElasticSearch connector would be helpful
[
https://issues.apache.org/jira/browse/CONNECTORS-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216851#comment-13216851
]
Karl Wright edited comment on CONNECTORS-288 at 2/26/12 9:16 PM:
-
Looking at the actual test run, the history reports the following at the end:
{code}
02-26-2012 16:09:25.129 Indexation (ElasticSearch)
http://localhost:9200/index/_optimize
OK 0 69
02-26-2012 16:09:24.939 job end 1330290457146(Test Job)
0 1
02-26-2012 16:09:14.909 Deletion (ElasticSearch)
http://localhost:9090/chemistry-opencmis-server-inmemory/atom...
/138/null
OK 0 7
02-26-2012 16:09:07.787 Optimize (ElasticSearch)
http://localhost:9090/chemistry-opencmis-server-inmemory/atom...
/140/null
OK 27 6
02-26-2012 16:09:07.778 Optimize (ElasticSearch)
http://localhost:9090/chemistry-opencmis-server-inmemory/atom...
/139/null
OK 27 7
02-26-2012 16:09:07.769 Optimize (ElasticSearch)
http://localhost:9090/chemistry-opencmis-server-inmemory/atom...
/137/null
OK 27 15
02-26-2012 16:09:05.278 job start 1330290457146(Test Job)
0 1
02-26-2012 16:08:55.020 Indexation (ElasticSearch)
http://localhost:9200/index/_optimize
OK 0 93
02-26-2012 16:08:54.926 job end 1330290457146(Test Job)
0 1
02-26-2012 16:08:47.678 Optimize (ElasticSearch)
http://localhost:9090/chemistry-opencmis-server-inmemory/atom...
/138/null
OK 27 10
02-26-2012 16:08:47.666 Optimize (ElasticSearch)
http://localhost:9090/chemistry-opencmis-server-inmemory/atom...
/140/null
OK 27 6
02-26-2012 16:08:47.652 Optimize (ElasticSearch)
http://localhost:9090/chemistry-opencmis-server-inmemory/atom...
/139/null
OK 27 11
02-26-2012 16:08:47.646 Optimize (ElasticSearch)
http://localhost:9090/chemistry-opencmis-server-inmemory/atom...
/137/null
OK 27 13
02-26-2012 16:08:45.192 job start 1330290457146(Test Job)
0 1
02-26-2012 16:08:34.940 Indexation (ElasticSearch)
http://localhost:9200/index/_optimize
OK 0 75
02-26-2012 16:08:34.917 job end 1330290457146(Test Job)
0 1
02-26-2012 16:08:29.502 Optimize (ElasticSearch)
http://localhost:9090/chemistry-opencmis-server-inmemory/atom...
/139/null
OK 27 10
02-26-2012 16:08:29.491 Optimize (ElasticSearch)
http://localhost:9090/chemistry-opencmis-server-inmemory/atom...
/137/null
OK 27 8
02-26-2012 16:08:29.412 Optimize (ElasticSearch)
http://localhost:9090/chemistry-opencmis-server-inmemory/atom...
/140/null
OK 27 66
02-26-2012 16:08:29.404 Optimize (ElasticSearch)
http://localhost:9090/chemistry-opencmis-server-inmemory/atom...
/138/null
OK 27 68
02-26-2012 16:08:25.097 job start 1330290457146(Test Job)
0 1
02-26-2012 16:08:24.846 Indexation (ElasticSearch)
http://localhost:9200/index/_optimize
OK 0 88
02-26-2012 16:08:14.890 job stop1330290457146(Test Job)
0 1
02-26-2012 16:08:09.041 Optimize (ElasticSearch)
http://localhost:9090/chemistry-opencmis-server-inmemory/atom...
/138/null
OK 27 868
02-26-2012 16:08:04.900 job start 1330290457146(Test Job)
0 1
{code}
The job at the end is stuck in the "Cleaning up" state, which indicates that it
is trying to delete the documents from the index, but is not succeeding for
some reason. The jobstatus reports 4 documents at that time.
The CMIS connector is not helping here because it does not seem to record ANY
activities. It also looks like the activities being recorded for the
ElasticSearch connector are backwards; it records "Optimize" when it should
record "Indexation", and visa versa.
was (Author: kwri...@metacarta.com):
Looking at the actual test run, the history reports the following at the
end:
{code}
02-26-2012 16:09:25.129 Indexation (ElasticSearch)
http://localhost:9200/index/_optimize
OK 0 69
02-26-2012 16:09:24.939 job end 1330290457146(Test Job)
0 1
02-26-2012 16:09:14.909 Deletion (ElasticSearch)
http://localhost:
[jira] [Issue Comment Edited] (CONNECTORS-288) An ElasticSearch connector would be helpful
[
https://issues.apache.org/jira/browse/CONNECTORS-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217519#comment-13217519
]
Karl Wright edited comment on CONNECTORS-288 at 2/27/12 8:25 PM:
-
bq. I tried to execute the test now and I think that now tests are runned
correctly but it seems that it can't delete the job from Manifold at the end of
the test:
Right, the problem is that the job deletion hangs, because it's trying to
delete the documents from the index and something goes wrong with that. I
posted earlier the manifoldcf.log output associated with this failure:
{code}
ERROR 2012-02-26 16:09:35,903 (Document delete thread '7') - Exception tossed:
Server/page not found
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Server/page not found
at
org.apache.manifoldcf.agents.output.elasticsearch.ElasticSearchConnection.call(ElasticSearchConnection.java:111)
at
org.apache.manifoldcf.agents.output.elasticsearch.ElasticSearchDelete.(ElasticSearchDelete.java:35)
at
org.apache.manifoldcf.agents.output.elasticsearch.ElasticSearchConnector.removeDocument(ElasticSearchConnector.java:378)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.removeDocument(IncrementalIngester.java:1598)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentDeleteMultiple(IncrementalIngester.java:748)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:130)
{code}
The issue is that the "Server/page not found" error seems to occur
intermittently on many different requests. These are usually retried, but at
the end during the delete phase the delete threads wait 5 minutes before
retrying, which is why the test fails, because it only waits 2 minutes. The
real problem is that we should not be getting these intermittent random errors
at all, which is why I think we need to look at data that is kept around in the
connector from request to request, namely the cached data structures. I am
certain these are the source of the problem.
was (Author: kwri...@metacarta.com):
bq. I tried to execute the test now and I think that now tests are runned
correctly but it seems that it can't delete the job from Manifold at the end of
the test:
Right, the problem is that the job deletion hangs, because it's trying to
delete the documents from the index and something goes wrong with that. I
posted earlier the manifoldcf.log output associated with this failure:
{code}
ERROR 2012-02-26 16:09:35,903 (Document delete thread '7') - Exception tossed:
Server/page not found
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Server/page not found
at
org.apache.manifoldcf.agents.output.elasticsearch.ElasticSearchConnection.call(ElasticSearchConnection.java:111)
at
org.apache.manifoldcf.agents.output.elasticsearch.ElasticSearchDelete.(ElasticSearchDelete.java:35)
at
org.apache.manifoldcf.agents.output.elasticsearch.ElasticSearchConnector.removeDocument(ElasticSearchConnector.java:378)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.removeDocument(IncrementalIngester.java:1598)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentDeleteMultiple(IncrementalIngester.java:748)
at
org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:130)
{code}
The issue is that the "Server/page not found" error seems to occur
intermittently on many different requests. These are usually retried, but at
the end during the delete phase they wait 5 minutes before being retried, which
is why the test fails. The real problem is that we should not be getting
intermittent random errors at all, which is why I think we need to look at data
that is kept around in the connector from request to request, namely the cached
data structures. I am certain these are the source of the problem.
> An ElasticSearch connector would be helpful
> ---
>
> Key: CONNECTORS-288
> URL: https://issues.apache.org/jira/browse/CONNECTORS-288
> Project: ManifoldCF
> Issue Type: New Feature
>Affects Versions: ManifoldCF 0.5
>Reporter: Piergiorgio Lucidi
>Assignee: Piergiorgio Lucidi
> Labels: elasticsearch
> Fix For: ManifoldCF next
>
> Attachments: manifold-elasticsearch-patch,
> manifold-elasticsearch-patch, manifold-elasticsearch-patch,
> manifold-elasticsearch-patch, manifold-elasticsearch-patch,
> manifold-elasticsearch-patch, manifold-elasticsearch-patch,
> manifold-elasticsearch-velocity-patch, manifoldcf-elasticsearch-project-patct
>
> Original Estimate: 120h
> Remaining Est
[jira] [Issue Comment Edited] (CONNECTORS-288) An ElasticSearch connector would be helpful
[ https://issues.apache.org/jira/browse/CONNECTORS-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221639#comment-13221639 ] Karl Wright edited comment on CONNECTORS-288 at 3/4/12 9:50 PM: Luca - I am away from internet access at the moment, but I will be back tomorrow. So I am having to relay this from memory. The test infrastructure works fine when I run it here. The stopAgentsRun flag is cleared when the agents process is shut down, I think. Nevertheless both Piergiorgio and myself have had no problem running the test and having it work fine except for the fact that it cannot delete documents. Can you do the following exactly as stated and let me know what happens for you: (1) Create a fresh checkout of https://svn.apache.org/repos/asf/incubator/lcf/branches/CONNECTORS-288 (2) Type "ant download-dependencies" (3) Type "ant run-elasticsearch-tests-derby" (4) Paste the results you get into a comment in this ticket. I'd like to do this to see whether you have the same setup I do. Thanks, Karl was (Author: kwri...@metacarta.com): Luca - I am away from internet access at the moment, but I will be back tomorrow. So I am having to relay this from memory. The test infrastructure works fine when I run it here. The stopAgentsRun flag is cleared when the agents process is shut down, I think. Nevertheless both Piergiorgio and myself have had no problem running the test and having it work fine except for the fact that it cannot delete documents. Can you do the following exactly as stated and let me know what happens for you: (1) Create a fresh checkout of https://svn.apache.org/repos/asf/incubator/lcf/branches/CONNECTORS-288 (2) Type "ant run-elasticsearch-tests-derby" (3) Paste the results you get into a comment in this ticket. Thanks, Karl > An ElasticSearch connector would be helpful > --- > > Key: CONNECTORS-288 > URL: https://issues.apache.org/jira/browse/CONNECTORS-288 > Project: ManifoldCF > Issue Type: New Feature >Affects Versions: ManifoldCF 0.5 >Reporter: Piergiorgio Lucidi >Assignee: Piergiorgio Lucidi > Labels: elasticsearch > Fix For: ManifoldCF next > > Attachments: manifold-elasticsearch-patch, > manifold-elasticsearch-patch, manifold-elasticsearch-patch, > manifold-elasticsearch-patch, manifold-elasticsearch-patch, > manifold-elasticsearch-patch, manifold-elasticsearch-patch, > manifold-elasticsearch-velocity-patch, manifoldcf-elasticsearch-project-patct > > Original Estimate: 120h > Remaining Estimate: 120h > > An ElasticSearch connector could be very useful to spread the use of > ManifoldCF -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-430) An error should be returned if invalid seeds are typed into the seeds list for the web connector
[
https://issues.apache.org/jira/browse/CONNECTORS-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239993#comment-13239993
]
Karl Wright edited comment on CONNECTORS-430 at 3/27/12 10:08 PM:
--
For me it fails:
{code}
[junit] Multipart posting url
'http://localhost:8346/mcf-crawler-ui/execute.jsp' with parameters
'outputname=MyOutputConnection&index=&recrawlinterval=1440&description=MyJob&startmethod=2&expirationinterval=&jobid=1332885945254&priority=5&reseedinterval=60&tabname=Connection&connectionname=MyRepositoryConnection&schedulerecords=0&scheduletype=1&type=job&op=Continue'
and 0 files...
[junit] Traceback (most recent call last):
[junit] File "test.py", line 166, in
[junit] var124.click()
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/VirtualBrowser.py",
line 120, in click
[junit] self.get_form( ).execute_javascript_expression( self.onclick )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/VirtualBrowser.py",
line 672, in execute_javascript_expression
[junit] return self.window_instance.execute_javascript_expression(
javascript )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/VirtualBrowser.py",
line 920, in execute_javascript_expression
[junit] return tokenstream.evaluate_expr( self.jscontext, "HTML" )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 1085, in evaluate_expr
[junit] rval = self.evaluate_expr1( context, place, parse_only )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 1120, in evaluate_expr1
[junit] rval = self.evaluate_expr2( context, place, parse_only )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 1164, in evaluate_expr2
[junit] return self.evaluate_expr3( context, place, parse_only )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 1176, in evaluate_expr3
[junit] rval = self.evaluate_expr4( context, place, parse_only )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 1264, in evaluate_expr4
[junit] rval = self.evaluate_expr5( context, place, parse_only )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 1299, in evaluate_expr5
[junit] rval = self.evaluate_expr6( context, place, parse_only )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 1341, in evaluate_expr6
[junit] return self.evaluate_expr7( context, place, parse_only )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 1438, in evaluate_expr7
[junit] return reference_object.call( arguments, context )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 554, in call
[junit] return self.get_referenced_object().call(argset,context)
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 186, in call
[junit] response = ts.evaluate_statement( context, "method %s" %
self.name )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 752, in evaluate_statement
[junit] result = self.evaluate_statement( newscope, place )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 988, in evaluate_statement
[junit] if self.evaluate_expr( context, place ) == None:
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 1085, in evaluate_expr
[junit] rval = self.evaluate_expr1( context, place, parse_only )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 1120, in evaluate_expr1
[junit] rval = self.evaluate_expr2( context, place, parse_only )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 1164, in evaluate_expr2
[junit] return self.evaluate_expr3( context, place, parse_only )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 1176, in evaluate_expr3
[junit] rval = self.evaluate_expr4( context, place, parse_only )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 1264, in evaluate_expr4
[junit] rval = self.evaluate_expr5( context, place, parse_only )
[junit] File
"/home/kwright/wip/trunk/tests/webcrawler/test-derby-output/Javascript.py",
line 1299, in evaluate_expr5
[junit]
[jira] [Issue Comment Edited] (CONNECTORS-460) ManifoldCF authority service doesn't handle multi-domain environments
[
https://issues.apache.org/jira/browse/CONNECTORS-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251493#comment-13251493
]
Karl Wright edited comment on CONNECTORS-460 at 4/11/12 12:06 PM:
--
Hi Colin,
The prefixes are not the issue. In ManifoldCF, there should be one authority
that is capable of dealing with all the interacting domains, just like
SharePoint understands all the domains by talking with just one domain
controller. An authority's job is to look up the SIDs for a given username,
and since your SharePoint instance can do this by talking with only one DC,
ManifoldCF's active directory authority connector ought to be able to do the
same thing.
The issue, then, is that the active directory authority connector doesn't know
how to deal with external, trusted domains. The authority must locate the
DC(s) for these domains within LDAP (I believe) and route requests to the
appropriate DC. Alternatively, if automatic discovery of the external DC is
too hard, we can modify the AD authority connector to simply allow the user to
describe multiple domain controllers.
Does this sound reasonable to you?
was (Author: kwri...@metacarta.com):
Hi Colin,
The prefixes are not the issue. In ManifoldCF, there should be one authority
that is capable of dealing with all the interacting domains, just like
SharePoint understands all the domains by talking with just one domain
controller. An authority's job is to look up the SIDs for a given username,
and since your SharePoint instance can do this by talking with only one DC,
ManifoldCF's active directory authority connector ought to be able to do the
same thing.
The issue, then, is that the active directory authority connector doesn't know
how to deal with external, trusted domains. The authority must located the
DC(s) for these domains within LDAP (I believe) and route requests to the
appropriate DC. Alternatively, if automatic discovery of the external DC is
too hard, we can modify the AD authority connector to simply allow the user to
describe multiple domain controllers.
Does this sound reasonable to you?
> ManifoldCF authority service doesn't handle multi-domain environments
> -
>
> Key: CONNECTORS-460
> URL: https://issues.apache.org/jira/browse/CONNECTORS-460
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Active Directory authority, Authority Service
> Environment: Two Active Directory domains: {{internal.com}} and
> {{external.com}}
> I'm indexing a Sharepoint site, where that site has permissions set
> from_both_domains
>Reporter: Colin Anderson
> Labels: active-directory, authorization, security
>
> The ManifoldCF authority service doesn't handle multi-domain environments.
> The authority service returns a list of SIDs for the specified user, from all
> available ManifoldCF authorities, for example:
> {{TOKEN:InternalAD:S-1-5-21-1234567890-1234567890-1234567890-1234}}
> Note that the SID is prefixed with the name of the ManifoldCF authority.
> Here is my setup:
> Output connector: Solr
> Authority connector1: Active Directory ({{internal.com}} domain), named
> {{InternalAD}}
> Authority connector2: Active Directory ({{external.com}} domain), named
> {{ExternalAD}}
> Repository connector: Sharepoint
> If I set the Sharepoint repository connector to use the authority 'None
> (Global Authority)', then {{allow_token_document}} will contain SIDs that are
> _not_ prefixed with any authority name, for example:
> {{S-1-5-21-1234567890-1234567890-1234567890-1234}}
> It is therefore not possible to get any search results, because the authority
> service tokens will not match the stored tokens (because they _are_ prefixed
> with authority names).
> If I set the Sharepoint repository connector to use one of the AD authorities
> 'InternalAD', then {{allow_token_document}} will contain SIDs that are
> prefixed with 'InternalAD', for example:
> {{TOKEN:InternalAD:S-1-5-21-1234567890-1234567890-1234567890-1234}}
> However, the prefix is _always_ 'InternalAD', even if the user/group actually
> belongs to the {{external.com}} domain. Therefore it is not possible for
> users in the {{external.com}} domain to get any search results, because the
> authority service tokens will not match the stored tokens.
> In essence, there seems to be a mismatch between the tokens that the
> authority service outputs, and those that repository connectors output.
> Perhaps one solution would be to use the authority 'None (Global Authority)',
> and modify the authority service to take an extra query parameter that
> prevents it from prefixing SIDs with the authority name.
--
This message is automatically
[jira] [Issue Comment Edited] (CONNECTORS-347) MySQL HopCount test sometimes fails
[
https://issues.apache.org/jira/browse/CONNECTORS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257899#comment-13257899
]
Karl Wright edited comment on CONNECTORS-347 at 4/19/12 11:23 PM:
--
Just saw it again; clearly it is "real".
My working hypothesis is that the counts returned by count(x) in the SQL
statements are in fact not synchronized with the transactions, so maybe the
count can "run behind". The context in which this happens is an initial crawl
of six documents, with a hopcount filter. If the filter is not working the
count would be 7; with a working filter, it's 6. But how can we then get 4?
was (Author: kwri...@metacarta.com):
Just saw it again; clearly it is "real".
> MySQL HopCount test sometimes fails
> ---
>
> Key: CONNECTORS-347
> URL: https://issues.apache.org/jira/browse/CONNECTORS-347
> Project: ManifoldCF
> Issue Type: Bug
> Components: Framework core
>Affects Versions: ManifoldCF 0.5
>Reporter: Karl Wright
>Assignee: Karl Wright
> Fix For: ManifoldCF next
>
>
> The HopcountMySQLIT test sometimes fails complaining that it is expecting to
> see 6 documents but only sees 4.
> {code}
> [junit] org.apache.manifoldcf.core.interfaces.ManifoldCFException: Wrong
> number of documents processed - expected 6, saw 4
> [junit] at
> org.apache.manifoldcf.filesystem_tests.HopcountMySQLIT.hopcountCheck(HopcountMySQLIT.java:152)
> {code}
> This does not happen on every run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-347) MySQL HopCount test sometimes fails
[
https://issues.apache.org/jira/browse/CONNECTORS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257899#comment-13257899
]
Karl Wright edited comment on CONNECTORS-347 at 4/20/12 1:42 AM:
-
Just saw it again; clearly it is "real".
My working hypothesis is that the counts returned by count() in the SQL
statements are in fact not synchronized with the transactions, so maybe the
count can "run behind". The context in which this happens is an initial crawl
of six documents, with a hopcount filter. If the filter is not working the
count would be 7; with a working filter, it's 6. But how can we then get 4?
was (Author: kwri...@metacarta.com):
Just saw it again; clearly it is "real".
My working hypothesis is that the counts returned by count(x) in the SQL
statements are in fact not synchronized with the transactions, so maybe the
count can "run behind". The context in which this happens is an initial crawl
of six documents, with a hopcount filter. If the filter is not working the
count would be 7; with a working filter, it's 6. But how can we then get 4?
> MySQL HopCount test sometimes fails
> ---
>
> Key: CONNECTORS-347
> URL: https://issues.apache.org/jira/browse/CONNECTORS-347
> Project: ManifoldCF
> Issue Type: Bug
> Components: Framework core
>Affects Versions: ManifoldCF 0.5, ManifoldCF 0.6
>Reporter: Karl Wright
>Assignee: Karl Wright
> Fix For: ManifoldCF 0.6
>
>
> The HopcountMySQLIT test sometimes fails complaining that it is expecting to
> see 6 documents but only sees 4.
> {code}
> [junit] org.apache.manifoldcf.core.interfaces.ManifoldCFException: Wrong
> number of documents processed - expected 6, saw 4
> [junit] at
> org.apache.manifoldcf.filesystem_tests.HopcountMySQLIT.hopcountCheck(HopcountMySQLIT.java:152)
> {code}
> This does not happen on every run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CONNECTORS-347) MySQL HopCount test sometimes fails
[
https://issues.apache.org/jira/browse/CONNECTORS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258202#comment-13258202
]
Karl Wright edited comment on CONNECTORS-347 at 4/20/12 12:58 PM:
--
Here's a postmortem for a run that succeeded:
{code}
mysql> select parentidhash,distance from hopcount;
+--+--+
| parentidhash | distance |
+--+--+
| 5432CFFCE5729BF1F3E695FFCF83B08E415694C6 |0 |
| 718564BC3D0C94B0F88B36B7FA2FC915F665A5E2 |3 |
| 1D9DB3181447F82677FB9AB91C56F33B1A2C75B0 |2 |
| B622AAB99E7BFF749E2C93E01009C3E02F90693C |2 |
| 9F6F59234CC36DF341184FAEC934BEEE516E1E60 |1 |
| CD31925B98C4B481F6523030A1C7C9D9DF0CC080 |1 |
| 1B26DBA4653B26B3151433EBB4C1CDCB8358C1F2 |1 |
+--+--+
7 rows in set (0.00 sec)
mysql> select parentidhash,childidhash from intrinsiclink;
+--+--+
| parentidhash | childidhash |
+--+--+
| 1B26DBA4653B26B3151433EBB4C1CDCB8358C1F2 |
5432CFFCE5729BF1F3E695FFCF83B08E415694C6 |
| 1D9DB3181447F82677FB9AB91C56F33B1A2C75B0 |
1B26DBA4653B26B3151433EBB4C1CDCB8358C1F2 |
| 5432CFFCE5729BF1F3E695FFCF83B08E415694C6 | |
| 718564BC3D0C94B0F88B36B7FA2FC915F665A5E2 |
1D9DB3181447F82677FB9AB91C56F33B1A2C75B0 |
| 9F6F59234CC36DF341184FAEC934BEEE516E1E60 |
5432CFFCE5729BF1F3E695FFCF83B08E415694C6 |
| B622AAB99E7BFF749E2C93E01009C3E02F90693C |
1B26DBA4653B26B3151433EBB4C1CDCB8358C1F2 |
| CD31925B98C4B481F6523030A1C7C9D9DF0CC080 |
5432CFFCE5729BF1F3E695FFCF83B08E415694C6 |
+--+--+
7 rows in set (0.00 sec)
mysql> select dochash,docid from jobqueue;
+--++
| dochash | docid
|
+--++
| 5432CFFCE5729BF1F3E695FFCF83B08E415694C6 |
C:\wip\mcf\trunk\tests\filesystem\test-mysql-output\testdata |
| B622AAB99E7BFF749E2C93E01009C3E02F90693C |
C:\wip\mcf\trunk\tests\filesystem\test-mysql-output\testdata\testdir\test3.txt |
| 1D9DB3181447F82677FB9AB91C56F33B1A2C75B0 |
C:\wip\mcf\trunk\tests\filesystem\test-mysql-output\testdata\testdir\seconddir |
| CD31925B98C4B481F6523030A1C7C9D9DF0CC080 |
C:\wip\mcf\trunk\tests\filesystem\test-mysql-output\testdata\test1.txt |
| 9F6F59234CC36DF341184FAEC934BEEE516E1E60 |
C:\wip\mcf\trunk\tests\filesystem\test-mysql-output\testdata\test2.txt |
| 1B26DBA4653B26B3151433EBB4C1CDCB8358C1F2 |
C:\wip\mcf\trunk\tests\filesystem\test-mysql-output\testdata\testdir |
+--++
6 rows in set (0.00 sec)
{code}
Even in this one, the jobqueue table is inconsistent with the hopcount and
intrinsiclink tables. They should all be consistent.
was (Author: kwri...@metacarta.com):
Here's a postmortem for a run that succeeded:
{code}
mysql> select parentidhash,distance from hopcount;
+--+--+
| parentidhash | distance |
+--+--+
| 5432CFFCE5729BF1F3E695FFCF83B08E415694C6 |0 |
| 718564BC3D0C94B0F88B36B7FA2FC915F665A5E2 |3 |
| 1D9DB3181447F82677FB9AB91C56F33B1A2C75B0 |2 |
| B622AAB99E7BFF749E2C93E01009C3E02F90693C |2 |
| 9F6F59234CC36DF341184FAEC934BEEE516E1E60 |1 |
| CD31925B98C4B481F6523030A1C7C9D9DF0CC080 |1 |
| 1B26DBA4653B26B3151433EBB4C1CDCB8358C1F2 |1 |
+--+--+
7 rows in set (0.00 sec)
mysql> select parentidhash,childidhash from intrinsiclink;
+--+--+
| parentidhash | childidhash |
+--+--+
| 1B26DBA4653B26B3151433EBB4C1CDCB8358C1F2 |
5432CFFCE5729BF1F3E695FFCF83B08E415694C6 |
| 1D9DB3181447F82677FB9AB91C56F33B1A2C75B0 |
1B26DBA4653B26B3151433EBB4C1CDCB8358C1F2 |
| 5432CFFCE5729BF1F3E695FFCF83B08E415694C6 | |
| 718564BC3D0C94B0F88B36B7FA2FC915F665A5E2 |
1D9DB3181447F82677FB9AB91C56F33B1A2C75B0 |
| 9F6F59234CC36DF341184FAEC934BEEE516E1E60 |
5432CFFCE5729BF1F3E6
[jira] [Issue Comment Edited] (CONNECTORS-347) MySQL HopCount test sometimes fails
[
https://issues.apache.org/jira/browse/CONNECTORS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258237#comment-13258237
]
Karl Wright edited comment on CONNECTORS-347 at 4/20/12 1:27 PM:
-
A postgresql run is as follows:
{code}
testdb=# select dochash,docid from jobqueue;
dochash | docid
--+-
EE6C0ED002CE9303183FEFEF111D6598203F6295 |
C:\wip\mcf\trunk\tests\filesystem\test-postgresql-output\testdata
E7DE0B8D3F153CECFD3C57B4D83E59073EC4DD60 |
C:\wip\mcf\trunk\tests\filesystem\test-postgresql-output\testdata\testdir
181B9C9FB7F62D24461967391B73FF7CB0B14CD9 |
C:\wip\mcf\trunk\tests\filesystem\test-postgresql-output\testdata\test2.txt
24081AC0B03B2394F38D3EB78A3BEDB6CCF1D981 |
C:\wip\mcf\trunk\tests\filesystem\test-postgresql-output\testdata\test1.txt
5AA0DB8B2B7D267FAD5E3E0E8926996DEFC4C666 |
C:\wip\mcf\trunk\tests\filesystem\test-postgresql-output\testdata\testdir\seconddir
4ED8AFCC917B604C1EE4C83458DB9B7265967F40 |
C:\wip\mcf\trunk\tests\filesystem\test-postgresql-output\testdata\testdir\test3.txt
(6 rows)
testdb=# select parentidhash, childidhash from intrinsiclink;
parentidhash | childidhash
--+--
EE6C0ED002CE9303183FEFEF111D6598203F6295 |
181B9C9FB7F62D24461967391B73FF7CB0B14CD9 |
EE6C0ED002CE9303183FEFEF111D6598203F6295
24081AC0B03B2394F38D3EB78A3BEDB6CCF1D981 |
EE6C0ED002CE9303183FEFEF111D6598203F6295
E7DE0B8D3F153CECFD3C57B4D83E59073EC4DD60 |
EE6C0ED002CE9303183FEFEF111D6598203F6295
4ED8AFCC917B604C1EE4C83458DB9B7265967F40 |
E7DE0B8D3F153CECFD3C57B4D83E59073EC4DD60
5AA0DB8B2B7D267FAD5E3E0E8926996DEFC4C666 |
E7DE0B8D3F153CECFD3C57B4D83E59073EC4DD60
7013D738BB77ED47763449997384046B20B77832 |
5AA0DB8B2B7D267FAD5E3E0E8926996DEFC4C666
(7 rows)
testdb=# select parentidhash, distance from hopcount;
parentidhash | distance
--+--
EE6C0ED002CE9303183FEFEF111D6598203F6295 |0
24081AC0B03B2394F38D3EB78A3BEDB6CCF1D981 |1
181B9C9FB7F62D24461967391B73FF7CB0B14CD9 |1
E7DE0B8D3F153CECFD3C57B4D83E59073EC4DD60 |1
5AA0DB8B2B7D267FAD5E3E0E8926996DEFC4C666 |2
4ED8AFCC917B604C1EE4C83458DB9B7265967F40 |2
7013D738BB77ED47763449997384046B20B77832 |3
(7 rows)
testdb=#
{code}
was (Author: kwri...@metacarta.com):
A postgresql run is as follows:
{code}
testdb=# select dochash,docid from jobqueue;
dochash |
docid
--+-
EE6C0ED002CE9303183FEFEF111D6598203F6295 | C:\wip\mcf\trunk\tests\filesystem\te
st-postgresql-output\testdata
E7DE0B8D3F153CECFD3C57B4D83E59073EC4DD60 | C:\wip\mcf\trunk\tests\filesystem\te
st-postgresql-output\testdata\testdir
181B9C9FB7F62D24461967391B73FF7CB0B14CD9 | C:\wip\mcf\trunk\tests\filesystem\te
st-postgresql-output\testdata\test2.txt
24081AC0B03B2394F38D3EB78A3BEDB6CCF1D981 | C:\wip\mcf\trunk\tests\filesystem\te
st-postgresql-output\testdata\test1.txt
5AA0DB8B2B7D267FAD5E3E0E8926996DEFC4C666 | C:\wip\mcf\trunk\tests\filesystem\te
st-postgresql-output\testdata\testdir\seconddir
4ED8AFCC917B604C1EE4C83458DB9B7265967F40 | C:\wip\mcf\trunk\tests\filesystem\te
st-postgresql-output\testdata\testdir\test3.txt
(6 rows)
testdb=# select parentidhash, childidhash from intrinsiclink;
parentidhash | childidhash
--+-
-
EE6C0ED002CE9303183FEFEF111D6598203F6295 |
181B9C9FB7F62D24461967391B73FF7CB0B14CD9 | EE6C0ED002CE9303183FEFEF111D6598203F
6295
24081AC0B03B2394F38D3EB78A3BEDB6CCF1D981 | EE6C0ED002CE9303183FEFEF111D6598203F
6295
E7DE0B8D3F153CECFD3C57B4D83E59073EC4DD60 | EE6C0ED002CE9303183FEFEF111D6598203F
6295
4ED8AFCC917B604C1EE4C83458DB9B7265967F40 | E7DE0B8D3F153CECFD3C57B4D83E59073EC4
DD60
5AA0DB8B2B7D267FAD5E3E0E8926996DEFC4C666 | E7DE0B8D3F153CECFD3C57B4D83E59073EC4
DD60
7013D738BB77ED47763449997384046B20B77832 | 5AA0DB8B2B7D267FAD5E3E0E8926996DEFC4
C666
(7 rows)
testdb=# select parentidhash, distance from hopcount;
parentidhash | distance
--+--
EE6C0ED002CE9303183FEFEF111D6598203F6295 |0
24081AC0B03B2394F38D3EB78A3BEDB6CCF1D981 |1
181B9C9FB7F62D24461967391B73FF7CB0B14CD9 |1
E7DE0B8D3F153CECFD3C57B4D83E59073EC4DD60 |1
5AA0DB8B2B7D267FAD5E3E0E8926996DEFC4C666 |2
4ED8AFCC917B604C1EE4C83458DB9B7265967F40
