[jira] Created: (CONNECTORS-50) Proposal for initial two releases of LCF, including packaged product and full API
Proposal for initial two releases of LCF, including packaged product and full API - Key: CONNECTORS-50 URL: https://issues.apache.org/jira/browse/CONNECTORS-50 Project: Lucene Connector Framework Issue Type: New Feature Components: Framework core Reporter: Jack Krupansky Currently, LCF has a relatively high-bar or evaluation and use, requiring developer expertise. Also, although LCF has a comprehensive UI, it is not currently packaged for use as a crawling engine for advanced applications. A small set of individual feature requests are needed to address these issues. They are summarized briefly to show how they fit together for two initial releases of LCF, but will be broken out into individual LCF Jira issues. Goals: 1. LCF as a standalone, downloadable, usable-out-of-the-box product (much as Solr is today) 2. LCF as a toolkit for developers needing customized crawling and repository access 3. An API-based crawling engine that can be integrated with applications (as Aperture is today) Larger goals: 1. Make it very easy for users to evaluate LCF. 2. Make it very easy for developers to customize LCF. 3. Make it very easy for appplications to fully manage and control LCF in operation. Two phases: 1) Standalone, packaged app that is super-easy to evaluate and deploy. Call it LCF 0.5. 2) API-based crawling engine for applications for which the UI might not be appropriate. Call it LCF 1.0. Phase 1 --- LCF 0.5 right out of the box would interface loosely with Solr 1.4 or later. It would contain roughly the features that are currently in place or currently underway, plus a little more. Specifically, LCF 0.5 would contain these additional capabilities: 1. Plug-in architecture for connectors (already underway) 2. Packaged app ready to run with embedded Jetty app server (I think this has been agreed to) 3. Bundled with database - PostgreSQL or derby - ready to run without additional manual setup 4. Mini-API to initially configure default connections and example jobs for file system and web crawl 5. Agent process started automatically (platform-specific startup required) 6. Solr output connector option to commit at end of job, by default Installation and basic evaluation of LCF would be essentially as simple as Solr is today. The example connections and jobs would permit the user to initiate example crawls of a file system example directory and an example web on the LCF web site with just a couple of clicks (as opposed to the detailed manual setup required today to create repository and output connections and jobs. It is worth considering whether the SharePoint connector could also be included as part of the default package. Users could then add additional connectors and repositories and jobs as desired. Timeframe for release? Level of effort? Phase 2 --- The essence of Phase 2 is that LCF would be split to allow direct, full API access to LCF as a crawling engine, in additional to the full LCF UI. Call this LCF 1.0. Specifically, LCF 1.0 would contain these additional capabilities: 1. Full API for LCF as a crawling engine 2. LCF can be bundled within an app (such as the default LCF package itself with its UI) 3. LCF event and activity notification for full control by an application (already a Jira request) Overall, LCF will offer roughly the same crawling capabilities as with LCF 0.5, plus whatever bug fixes and minor enhancements might also be added. Timeframe for release? Level of effort? - Issues: - Can we package PostgreSQL with LCF so LCF can set it up? - Or do we need Derby for that purpose? - Managing multiple processes (UI, database, agent, app processes) - What exactly would the API look like? (URL, XML, JSON, YAML?) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CONNECTORS-47) Framework UI seems to call connector post processing more than needed
[ https://issues.apache.org/jira/browse/CONNECTORS-47?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-47. --- Assignee: Karl Wright Resolution: Fixed r959393. Refactor as needed to solidify the contract between edit pages and the execute.jsp post page. Framework UI seems to call connector post processing more than needed - Key: CONNECTORS-47 URL: https://issues.apache.org/jira/browse/CONNECTORS-47 Project: Lucene Connector Framework Issue Type: Bug Components: Framework crawler agent Reporter: Karl Wright Assignee: Karl Wright Priority: Minor Connector form post processing is currently invoked both in execute.jsp (which is the target of all form posts), as well as in individual edit pages (such as editconfig.jsp and editjob.jsp). Unless a reason can be found for why this is done, the individual edit page calls should be removed, since they are by definition superfluous. Possible reasons it was done this way were: (a) that code predates execute.jsp (b) some other functionality, e.g. copy or posting of certificates, needs it At any rate, this should be looked at after the bulk of CONNECTORS-40 related changes are committed to trunk. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (CONNECTORS-51) Reduce the number of required -D defines by using System.setProperty() in the appropriate places
Reduce the number of required -D defines by using System.setProperty() in the appropriate places Key: CONNECTORS-51 URL: https://issues.apache.org/jira/browse/CONNECTORS-51 Project: Lucene Connector Framework Issue Type: Improvement Components: JCIFS connector Reporter: Karl Wright Priority: Minor The JCIFS connector requires a fair number of -D switches in the java startup in order to do the right things. This is largely because jcifs.jar is constructed this way. It may be possible, however, to eliminate these -D's by judicious static use of System.setProperty() within the appropriate connector class, provided we presume that jcifs classes will never be loaded prior to the jcifs connector classes being loaded. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CONNECTORS-40) Classloader-based plug-in architecture would permit LCF to be prebuilt
[ https://issues.apache.org/jira/browse/CONNECTORS-40?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright reassigned CONNECTORS-40: - Assignee: Karl Wright Classloader-based plug-in architecture would permit LCF to be prebuilt -- Key: CONNECTORS-40 URL: https://issues.apache.org/jira/browse/CONNECTORS-40 Project: Lucene Connector Framework Issue Type: Improvement Components: Framework core Reporter: Karl Wright Assignee: Karl Wright The LCF architecture at this point requires interaction with the build script in order to add connectors. This is because the connector JSPs and jars need to be added to the appropriate war files. However, there is another architectural option that would eliminate this need, which is to use a custom classloader to pull components from jars that are placed in a specific directory or directories. In order for this to work, however, the UI components of every connector must become part of a jar. That implies that they will need to cease being JSPs, and become instead methods of each connector class. (There is no proscription against using something like Velocity for assembling the necessary output for a connector, however.) Limiting the backwards-compatibility impact of this change will be difficult, especially after a first release is made, so it seems clear that any change along these lines should be attempted before version 1.0 is released. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
