Some more thoughts on a classloader plug-in style architecture
It occurred to me that a classloader plug-in reader for LCF would not achieve the goal of allowing a fully prebuilt LCF with connector add-ons. The reason, which should have been obvious from the beginning, is because each connector consists not only of the Java implementation, but also a UI component. The UI component will need a mechanism similar to the classloader one in order for everything to work. It is possible, I suppose, for precompiled JSP's to be class-loaded instead of uncompiled JSP's in the lcf-crawler-ui.war file. However this is going to require some care and finesse (and example build.xml files) to get it to work properly. Karl
Re: Some more thoughts on a classloader plug-in style architecture
Good point. LCF is a bit more complex than Solr in that sense. Maybe a separate class is needed that has methods to retrieve the crawl and UI components of a connector. Or a small XML file with whatever info about the connector is needed. Or maybe it is simple enough for a properties file. Or maybe just a naming convention so that the name of the UI component can be deduced given the logical name of a connector. -- Jack Krupansky -- From: karl.wri...@nokia.com Sent: Wednesday, June 02, 2010 6:45 AM To: connectors-dev@incubator.apache.org Subject: Some more thoughts on a classloader plug-in style architecture It occurred to me that a classloader plug-in reader for LCF would not achieve the goal of allowing a fully prebuilt LCF with connector add-ons. The reason, which should have been obvious from the beginning, is because each connector consists not only of the Java implementation, but also a UI component. The UI component will need a mechanism similar to the classloader one in order for everything to work. It is possible, I suppose, for precompiled JSP's to be class-loaded instead of uncompiled JSP's in the lcf-crawler-ui.war file. However this is going to require some care and finesse (and example build.xml files) to get it to work properly. Karl
RE: Some more thoughts on a classloader plug-in style architecture
A naming convention is what is used now. This is described in the documentation. The problem is complicated because a precompiled JSP class will not fly - the package name is going to be different from application server to application server. So, either: - we abandon the thought of classloader-enabled plugin architecture, or - we recast the connector interface so that it uses something other than JSP for the plug-in contributions to the UI The latter would consist of some method that would be able to write to the response object output stream, and would have access to variables and multipart post data, but would otherwise be just a standard method. That is, there would be no JSP help whatsoever. The other downside is that it's not clear how exactly errors would be dealt with - the current convention is to use jsp:forward tags, and there would be no equivalent of that. Alternatively, if someone knows of a classloader-like construct present in all application servers, which could be overridden to load additional web application components, please let me know. I doubt very much there is such a beast, but you learn something new every day and maybe this would be one of them. Karl -Original Message- From: ext Jack Krupansky [mailto:jack.krupan...@lucidimagination.com] Sent: Wednesday, June 02, 2010 6:56 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Good point. LCF is a bit more complex than Solr in that sense. Maybe a separate class is needed that has methods to retrieve the crawl and UI components of a connector. Or a small XML file with whatever info about the connector is needed. Or maybe it is simple enough for a properties file. Or maybe just a naming convention so that the name of the UI component can be deduced given the logical name of a connector. -- Jack Krupansky -- From: karl.wri...@nokia.com Sent: Wednesday, June 02, 2010 6:45 AM To: connectors-dev@incubator.apache.org Subject: Some more thoughts on a classloader plug-in style architecture It occurred to me that a classloader plug-in reader for LCF would not achieve the goal of allowing a fully prebuilt LCF with connector add-ons. The reason, which should have been obvious from the beginning, is because each connector consists not only of the Java implementation, but also a UI component. The UI component will need a mechanism similar to the classloader one in order for everything to work. It is possible, I suppose, for precompiled JSP's to be class-loaded instead of uncompiled JSP's in the lcf-crawler-ui.war file. However this is going to require some care and finesse (and example build.xml files) to get it to work properly. Karl
RE: Some more thoughts on a classloader plug-in style architecture
The observation about JSPs is exactly what I was alluding to. Does the entire UI have to be converted to Velocity for this approach to work? There's an intermediate path that would involve converting only the connector portions, which might be viable. Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 7:27 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Actually the problem is quite tractable. We switch the UI over to Velocity templates (like Solr's VelocityResponseWriter, for example) and embed the UI bits into plugin JAR files that can live externally. JSPs aren't really workable in this fashion. Velocity templates can be loaded from the file system, the classpath, from a String, or from thin air. For example, VelocityResponseWriter allows templates to load from the actual request (clients can send in a template as an HTTP parameter), if not found it looks on the filesystem, and if not found it looks in the classpath. Erik On Jun 2, 2010, at 6:55 AM, Jack Krupansky wrote: Good point. LCF is a bit more complex than Solr in that sense. Maybe a separate class is needed that has methods to retrieve the crawl and UI components of a connector. Or a small XML file with whatever info about the connector is needed. Or maybe it is simple enough for a properties file. Or maybe just a naming convention so that the name of the UI component can be deduced given the logical name of a connector. -- Jack Krupansky -- From: karl.wri...@nokia.com Sent: Wednesday, June 02, 2010 6:45 AM To: connectors-dev@incubator.apache.org Subject: Some more thoughts on a classloader plug-in style architecture It occurred to me that a classloader plug-in reader for LCF would not achieve the goal of allowing a fully prebuilt LCF with connector add-ons. The reason, which should have been obvious from the beginning, is because each connector consists not only of the Java implementation, but also a UI component. The UI component will need a mechanism similar to the classloader one in order for everything to work. It is possible, I suppose, for precompiled JSP's to be class-loaded instead of uncompiled JSP's in the lcf-crawler-ui.war file. However this is going to require some care and finesse (and example build.xml files) to get it to work properly. Karl
Re: Some more thoughts on a classloader plug-in style architecture
Velocity is just a simple templating engine, so it could be used in the intermediate fashion to produce only snippets that mesh into the rest of the built-in UI, no problem. Erik On Jun 2, 2010, at 8:04 AM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: Does the entire UI have to be converted to Velocity for this approach to work? There's an intermediate path that would involve converting only the connector portions, which might be viable. Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 7:27 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Actually the problem is quite tractable. We switch the UI over to Velocity templates (like Solr's VelocityResponseWriter, for example) and embed the UI bits into plugin JAR files that can live externally. JSPs aren't really workable in this fashion. Velocity templates can be loaded from the file system, the classpath, from a String, or from thin air. For example, VelocityResponseWriter allows templates to load from the actual request (clients can send in a template as an HTTP parameter), if not found it looks on the filesystem, and if not found it looks in the classpath. Erik On Jun 2, 2010, at 6:55 AM, Jack Krupansky wrote: Good point. LCF is a bit more complex than Solr in that sense. Maybe a separate class is needed that has methods to retrieve the crawl and UI components of a connector. Or a small XML file with whatever info about the connector is needed. Or maybe it is simple enough for a properties file. Or maybe just a naming convention so that the name of the UI component can be deduced given the logical name of a connector. -- Jack Krupansky -- From: karl.wri...@nokia.com Sent: Wednesday, June 02, 2010 6:45 AM To: connectors-dev@incubator.apache.org Subject: Some more thoughts on a classloader plug-in style architecture It occurred to me that a classloader plug-in reader for LCF would not achieve the goal of allowing a fully prebuilt LCF with connector add-ons. The reason, which should have been obvious from the beginning, is because each connector consists not only of the Java implementation, but also a UI component. The UI component will need a mechanism similar to the classloader one in order for everything to work. It is possible, I suppose, for precompiled JSP's to be class-loaded instead of uncompiled JSP's in the lcf-crawler-ui.war file. However this is going to require some care and finesse (and example build.xml files) to get it to work properly. Karl
RE: Some more thoughts on a classloader plug-in style architecture
I've just spelunked through what I could find online, and it seems at least plausible to use Velocity for various LCF HTML templating needs. The major concern that I have is that the mix of inline java to HTML in the LCF stuff is weighted heavily towards inline java - which doesn't seem to be where Velocity's sweet spot is. ;-) The general underlying idea of invoking a connector-API-based method instead of providing a JSP seems sound, however, no matter what the templating engine is that does the final assembly. Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 8:19 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Velocity is just a simple templating engine, so it could be used in the intermediate fashion to produce only snippets that mesh into the rest of the built-in UI, no problem. Erik On Jun 2, 2010, at 8:04 AM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: Does the entire UI have to be converted to Velocity for this approach to work? There's an intermediate path that would involve converting only the connector portions, which might be viable. Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 7:27 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Actually the problem is quite tractable. We switch the UI over to Velocity templates (like Solr's VelocityResponseWriter, for example) and embed the UI bits into plugin JAR files that can live externally. JSPs aren't really workable in this fashion. Velocity templates can be loaded from the file system, the classpath, from a String, or from thin air. For example, VelocityResponseWriter allows templates to load from the actual request (clients can send in a template as an HTTP parameter), if not found it looks on the filesystem, and if not found it looks in the classpath. Erik On Jun 2, 2010, at 6:55 AM, Jack Krupansky wrote: Good point. LCF is a bit more complex than Solr in that sense. Maybe a separate class is needed that has methods to retrieve the crawl and UI components of a connector. Or a small XML file with whatever info about the connector is needed. Or maybe it is simple enough for a properties file. Or maybe just a naming convention so that the name of the UI component can be deduced given the logical name of a connector. -- Jack Krupansky -- From: karl.wri...@nokia.com Sent: Wednesday, June 02, 2010 6:45 AM To: connectors-dev@incubator.apache.org Subject: Some more thoughts on a classloader plug-in style architecture It occurred to me that a classloader plug-in reader for LCF would not achieve the goal of allowing a fully prebuilt LCF with connector add-ons. The reason, which should have been obvious from the beginning, is because each connector consists not only of the Java implementation, but also a UI component. The UI component will need a mechanism similar to the classloader one in order for everything to work. It is possible, I suppose, for precompiled JSP's to be class-loaded instead of uncompiled JSP's in the lcf-crawler-ui.war file. However this is going to require some care and finesse (and example build.xml files) to get it to work properly. Karl
[jira] Created: (CONNECTORS-40) Classloader-based plug-in architecture would permit LCF to be prebuilt
Classloader-based plug-in architecture would permit LCF to be prebuilt -- Key: CONNECTORS-40 URL: https://issues.apache.org/jira/browse/CONNECTORS-40 Project: Lucene Connector Framework Issue Type: Improvement Components: Framework core Reporter: Karl Wright The LCF architecture at this point requires interaction with the build script in order to add connectors. This is because the connector JSPs and jars need to be added to the appropriate war files. However, there is another architectural option that would eliminate this need, which is to use a custom classloader to pull components from jars that are placed in a specific directory or directories. In order for this to work, however, the UI components of every connector must become part of a jar. That implies that they will need to cease being JSPs, and become instead methods of each connector class. (There is no proscription against using something like Velocity for assembling the necessary output for a connector, however.) Limiting the backwards-compatibility impact of this change will be difficult, especially after a first release is made, so it seems clear that any change along these lines should be attempted before version 1.0 is released. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Some more thoughts on a classloader plug-in style architecture
Yeah, definitely embedded Java (ala JSP's %% stuff) isn't how Velocity works. It can call Java code just fine via tools (Java objects) that are injected into the Velocity context. Any sophisticated business logic can be distilled from the existing plugin JSPs and migrated to Java classes and added to the Velocity context. If specific connectors need more logic, they could ship with their own tools, and we build in a mechanism to pull in custom tools into the context. Erik On Jun 2, 2010, at 8:28 AM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: I've just spelunked through what I could find online, and it seems at least plausible to use Velocity for various LCF HTML templating needs. The major concern that I have is that the mix of inline java to HTML in the LCF stuff is weighted heavily towards inline java - which doesn't seem to be where Velocity's sweet spot is. ;-) The general underlying idea of invoking a connector-API-based method instead of providing a JSP seems sound, however, no matter what the templating engine is that does the final assembly. Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 8:19 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Velocity is just a simple templating engine, so it could be used in the intermediate fashion to produce only snippets that mesh into the rest of the built-in UI, no problem. Erik On Jun 2, 2010, at 8:04 AM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: Does the entire UI have to be converted to Velocity for this approach to work? There's an intermediate path that would involve converting only the connector portions, which might be viable. Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 7:27 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Actually the problem is quite tractable. We switch the UI over to Velocity templates (like Solr's VelocityResponseWriter, for example) and embed the UI bits into plugin JAR files that can live externally. JSPs aren't really workable in this fashion. Velocity templates can be loaded from the file system, the classpath, from a String, or from thin air. For example, VelocityResponseWriter allows templates to load from the actual request (clients can send in a template as an HTTP parameter), if not found it looks on the filesystem, and if not found it looks in the classpath. Erik On Jun 2, 2010, at 6:55 AM, Jack Krupansky wrote: Good point. LCF is a bit more complex than Solr in that sense. Maybe a separate class is needed that has methods to retrieve the crawl and UI components of a connector. Or a small XML file with whatever info about the connector is needed. Or maybe it is simple enough for a properties file. Or maybe just a naming convention so that the name of the UI component can be deduced given the logical name of a connector. -- Jack Krupansky -- From: karl.wri...@nokia.com Sent: Wednesday, June 02, 2010 6:45 AM To: connectors-dev@incubator.apache.org Subject: Some more thoughts on a classloader plug-in style architecture It occurred to me that a classloader plug-in reader for LCF would not achieve the goal of allowing a fully prebuilt LCF with connector add-ons. The reason, which should have been obvious from the beginning, is because each connector consists not only of the Java implementation, but also a UI component. The UI component will need a mechanism similar to the classloader one in order for everything to work. It is possible, I suppose, for precompiled JSP's to be class-loaded instead of uncompiled JSP's in the lcf-crawler-ui.war file. However this is going to require some care and finesse (and example build.xml files) to get it to work properly. Karl
RE: Some more thoughts on a classloader plug-in style architecture
I've entered a ticket CONNECTORS-40 for this work. What I propose is that this gets done before first official LCF release, because of the potential backwards-compatibility issues involved. It is, however, quite a heavy lift - I can't imagine getting it done in less than a couple of weeks straight-out effort. To minimize the elapsed development time of this step, I propose that we not attempt to convert any JSP's from the framework itself in this round, but merely those for each connector. I'll start by adding appropriate placeholder methods to the interfaces and all connectors, and then try to port the UI components one at a time, for one specific connector. Then, we can have a back-and-forth about how to refine that one implementation before I attempt the rest. Sound reasonable? Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 8:51 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Yeah, definitely embedded Java (ala JSP's %% stuff) isn't how Velocity works. It can call Java code just fine via tools (Java objects) that are injected into the Velocity context. Any sophisticated business logic can be distilled from the existing plugin JSPs and migrated to Java classes and added to the Velocity context. If specific connectors need more logic, they could ship with their own tools, and we build in a mechanism to pull in custom tools into the context. Erik On Jun 2, 2010, at 8:28 AM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: I've just spelunked through what I could find online, and it seems at least plausible to use Velocity for various LCF HTML templating needs. The major concern that I have is that the mix of inline java to HTML in the LCF stuff is weighted heavily towards inline java - which doesn't seem to be where Velocity's sweet spot is. ;-) The general underlying idea of invoking a connector-API-based method instead of providing a JSP seems sound, however, no matter what the templating engine is that does the final assembly. Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 8:19 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Velocity is just a simple templating engine, so it could be used in the intermediate fashion to produce only snippets that mesh into the rest of the built-in UI, no problem. Erik On Jun 2, 2010, at 8:04 AM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: Does the entire UI have to be converted to Velocity for this approach to work? There's an intermediate path that would involve converting only the connector portions, which might be viable. Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 7:27 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Actually the problem is quite tractable. We switch the UI over to Velocity templates (like Solr's VelocityResponseWriter, for example) and embed the UI bits into plugin JAR files that can live externally. JSPs aren't really workable in this fashion. Velocity templates can be loaded from the file system, the classpath, from a String, or from thin air. For example, VelocityResponseWriter allows templates to load from the actual request (clients can send in a template as an HTTP parameter), if not found it looks on the filesystem, and if not found it looks in the classpath. Erik On Jun 2, 2010, at 6:55 AM, Jack Krupansky wrote: Good point. LCF is a bit more complex than Solr in that sense. Maybe a separate class is needed that has methods to retrieve the crawl and UI components of a connector. Or a small XML file with whatever info about the connector is needed. Or maybe it is simple enough for a properties file. Or maybe just a naming convention so that the name of the UI component can be deduced given the logical name of a connector. -- Jack Krupansky -- From: karl.wri...@nokia.com Sent: Wednesday, June 02, 2010 6:45 AM To: connectors-dev@incubator.apache.org Subject: Some more thoughts on a classloader plug-in style architecture It occurred to me that a classloader plug-in reader for LCF would not achieve the goal of allowing a fully prebuilt LCF with connector add-ons. The reason, which should have been obvious from the beginning, is because each connector consists not only of the Java implementation, but also a UI component. The UI component will need a mechanism similar to the classloader one in order for everything to work. It is possible, I suppose, for precompiled JSP's to be class-loaded instead of
Re: Some more thoughts on a classloader plug-in style architecture
Totally reasonable. We'll try to devote some cycles to this too, of course, but probably not for a couple of weeks. Erik On Jun 2, 2010, at 9:00 AM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: I've entered a ticket CONNECTORS-40 for this work. What I propose is that this gets done before first official LCF release, because of the potential backwards-compatibility issues involved. It is, however, quite a heavy lift - I can't imagine getting it done in less than a couple of weeks straight-out effort. To minimize the elapsed development time of this step, I propose that we not attempt to convert any JSP's from the framework itself in this round, but merely those for each connector. I'll start by adding appropriate placeholder methods to the interfaces and all connectors, and then try to port the UI components one at a time, for one specific connector. Then, we can have a back-and-forth about how to refine that one implementation before I attempt the rest. Sound reasonable? Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 8:51 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Yeah, definitely embedded Java (ala JSP's %% stuff) isn't how Velocity works. It can call Java code just fine via tools (Java objects) that are injected into the Velocity context. Any sophisticated business logic can be distilled from the existing plugin JSPs and migrated to Java classes and added to the Velocity context. If specific connectors need more logic, they could ship with their own tools, and we build in a mechanism to pull in custom tools into the context. Erik On Jun 2, 2010, at 8:28 AM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: I've just spelunked through what I could find online, and it seems at least plausible to use Velocity for various LCF HTML templating needs. The major concern that I have is that the mix of inline java to HTML in the LCF stuff is weighted heavily towards inline java - which doesn't seem to be where Velocity's sweet spot is. ;-) The general underlying idea of invoking a connector-API-based method instead of providing a JSP seems sound, however, no matter what the templating engine is that does the final assembly. Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 8:19 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Velocity is just a simple templating engine, so it could be used in the intermediate fashion to produce only snippets that mesh into the rest of the built-in UI, no problem. Erik On Jun 2, 2010, at 8:04 AM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: Does the entire UI have to be converted to Velocity for this approach to work? There's an intermediate path that would involve converting only the connector portions, which might be viable. Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 7:27 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Actually the problem is quite tractable. We switch the UI over to Velocity templates (like Solr's VelocityResponseWriter, for example) and embed the UI bits into plugin JAR files that can live externally. JSPs aren't really workable in this fashion. Velocity templates can be loaded from the file system, the classpath, from a String, or from thin air. For example, VelocityResponseWriter allows templates to load from the actual request (clients can send in a template as an HTTP parameter), if not found it looks on the filesystem, and if not found it looks in the classpath. Erik On Jun 2, 2010, at 6:55 AM, Jack Krupansky wrote: Good point. LCF is a bit more complex than Solr in that sense. Maybe a separate class is needed that has methods to retrieve the crawl and UI components of a connector. Or a small XML file with whatever info about the connector is needed. Or maybe it is simple enough for a properties file. Or maybe just a naming convention so that the name of the UI component can be deduced given the logical name of a connector. -- Jack Krupansky -- From: karl.wri...@nokia.com Sent: Wednesday, June 02, 2010 6:45 AM To: connectors-dev@incubator.apache.org Subject: Some more thoughts on a classloader plug-in style architecture It occurred to me that a classloader plug-in reader for LCF would not achieve the goal of allowing a fully prebuilt LCF with connector add-ons. The reason, which should have been obvious from the beginning, is because each connector consists not only of the Java implementation, but also a UI component. The UI component will
[jira] Created: (CONNECTORS-41) Add hooks to output connectors for receiving event notifications, specifically job start, job end, etc.
Add hooks to output connectors for receiving event notifications, specifically job start, job end, etc. --- Key: CONNECTORS-41 URL: https://issues.apache.org/jira/browse/CONNECTORS-41 Project: Lucene Connector Framework Issue Type: Improvement Components: Framework core Reporter: Karl Wright Priority: Minor Currently there is no logic that informs an output connection of a job start, end, deletion, or other activity. While this would seem to have little to do with an output connector, this feature has been requested by Jack Krupansky as a potential way of deciding when to tell Solr to commit documents, rather than leave it up to Solr's configuration. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Some more thoughts on a classloader plug-in style architecture
Sounds good to me, assuming that LCF remains relatively stable. I am presuming that a fair number of people can and will be using LCF for various purposes well before actual formal first release anyway. The point being that delaying the formal release shouldn't slow most people from using LCF as is, provided that any phases of instability are relatively contained, which seems to be the normal case here anyway. -- Jack Krupansky -- From: karl.wri...@nokia.com Sent: Wednesday, June 02, 2010 9:00 AM To: connectors-dev@incubator.apache.org Subject: RE: Some more thoughts on a classloader plug-in style architecture I've entered a ticket CONNECTORS-40 for this work. What I propose is that this gets done before first official LCF release, because of the potential backwards-compatibility issues involved. It is, however, quite a heavy lift - I can't imagine getting it done in less than a couple of weeks straight-out effort. To minimize the elapsed development time of this step, I propose that we not attempt to convert any JSP's from the framework itself in this round, but merely those for each connector. I'll start by adding appropriate placeholder methods to the interfaces and all connectors, and then try to port the UI components one at a time, for one specific connector. Then, we can have a back-and-forth about how to refine that one implementation before I attempt the rest. Sound reasonable? Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 8:51 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Yeah, definitely embedded Java (ala JSP's %% stuff) isn't how Velocity works. It can call Java code just fine via tools (Java objects) that are injected into the Velocity context. Any sophisticated business logic can be distilled from the existing plugin JSPs and migrated to Java classes and added to the Velocity context. If specific connectors need more logic, they could ship with their own tools, and we build in a mechanism to pull in custom tools into the context. Erik On Jun 2, 2010, at 8:28 AM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: I've just spelunked through what I could find online, and it seems at least plausible to use Velocity for various LCF HTML templating needs. The major concern that I have is that the mix of inline java to HTML in the LCF stuff is weighted heavily towards inline java - which doesn't seem to be where Velocity's sweet spot is. ;-) The general underlying idea of invoking a connector-API-based method instead of providing a JSP seems sound, however, no matter what the templating engine is that does the final assembly. Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 8:19 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Velocity is just a simple templating engine, so it could be used in the intermediate fashion to produce only snippets that mesh into the rest of the built-in UI, no problem. Erik On Jun 2, 2010, at 8:04 AM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: Does the entire UI have to be converted to Velocity for this approach to work? There's an intermediate path that would involve converting only the connector portions, which might be viable. Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 7:27 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Actually the problem is quite tractable. We switch the UI over to Velocity templates (like Solr's VelocityResponseWriter, for example) and embed the UI bits into plugin JAR files that can live externally. JSPs aren't really workable in this fashion. Velocity templates can be loaded from the file system, the classpath, from a String, or from thin air. For example, VelocityResponseWriter allows templates to load from the actual request (clients can send in a template as an HTTP parameter), if not found it looks on the filesystem, and if not found it looks in the classpath. Erik On Jun 2, 2010, at 6:55 AM, Jack Krupansky wrote: Good point. LCF is a bit more complex than Solr in that sense. Maybe a separate class is needed that has methods to retrieve the crawl and UI components of a connector. Or a small XML file with whatever info about the connector is needed. Or maybe it is simple enough for a properties file. Or maybe just a naming convention so that the name of the UI component can be deduced given the logical name of a connector. -- Jack Krupansky -- From: karl.wri...@nokia.com Sent: Wednesday, June 02, 2010 6:45 AM To:
RE: Some more thoughts on a classloader plug-in style architecture
People who write connectors for LCF before this change occurs will need to revamp them completely afterwards. That is, unless they contribute them to the LCF code stream before the work begins... Second, since there is no Q/A infrastructure yet, there's a very good chance that stuff will get busted by such a massive change and go undetected. I would therefore greatly prefer that Q/A infrastructure get created before such changes, but that is not something I have any control over, unfortunately. I'm working as hard as I can towards having at least some core framework tests in place, but the UI testing will necessarily involve many instances of proprietary third-party repositories. Karl -Original Message- From: ext Jack Krupansky [mailto:jack.krupan...@lucidimagination.com] Sent: Wednesday, June 02, 2010 11:06 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Sounds good to me, assuming that LCF remains relatively stable. I am presuming that a fair number of people can and will be using LCF for various purposes well before actual formal first release anyway. The point being that delaying the formal release shouldn't slow most people from using LCF as is, provided that any phases of instability are relatively contained, which seems to be the normal case here anyway. -- Jack Krupansky -- From: karl.wri...@nokia.com Sent: Wednesday, June 02, 2010 9:00 AM To: connectors-dev@incubator.apache.org Subject: RE: Some more thoughts on a classloader plug-in style architecture I've entered a ticket CONNECTORS-40 for this work. What I propose is that this gets done before first official LCF release, because of the potential backwards-compatibility issues involved. It is, however, quite a heavy lift - I can't imagine getting it done in less than a couple of weeks straight-out effort. To minimize the elapsed development time of this step, I propose that we not attempt to convert any JSP's from the framework itself in this round, but merely those for each connector. I'll start by adding appropriate placeholder methods to the interfaces and all connectors, and then try to port the UI components one at a time, for one specific connector. Then, we can have a back-and-forth about how to refine that one implementation before I attempt the rest. Sound reasonable? Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 8:51 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Yeah, definitely embedded Java (ala JSP's %% stuff) isn't how Velocity works. It can call Java code just fine via tools (Java objects) that are injected into the Velocity context. Any sophisticated business logic can be distilled from the existing plugin JSPs and migrated to Java classes and added to the Velocity context. If specific connectors need more logic, they could ship with their own tools, and we build in a mechanism to pull in custom tools into the context. Erik On Jun 2, 2010, at 8:28 AM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: I've just spelunked through what I could find online, and it seems at least plausible to use Velocity for various LCF HTML templating needs. The major concern that I have is that the mix of inline java to HTML in the LCF stuff is weighted heavily towards inline java - which doesn't seem to be where Velocity's sweet spot is. ;-) The general underlying idea of invoking a connector-API-based method instead of providing a JSP seems sound, however, no matter what the templating engine is that does the final assembly. Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 8:19 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Velocity is just a simple templating engine, so it could be used in the intermediate fashion to produce only snippets that mesh into the rest of the built-in UI, no problem. Erik On Jun 2, 2010, at 8:04 AM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: Does the entire UI have to be converted to Velocity for this approach to work? There's an intermediate path that would involve converting only the connector portions, which might be viable. Karl -Original Message- From: ext Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 02, 2010 7:27 AM To: connectors-dev@incubator.apache.org Subject: Re: Some more thoughts on a classloader plug-in style architecture Actually the problem is quite tractable. We switch the UI over to Velocity templates (like Solr's VelocityResponseWriter, for example) and embed the UI bits into plugin JAR files that can live externally.
[jira] Resolved: (CONNECTORS-39) Database abstraction layer does not abstract from transactions
[ https://issues.apache.org/jira/browse/CONNECTORS-39?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-39. --- Resolution: Fixed Database abstraction layer does not abstract from transactions -- Key: CONNECTORS-39 URL: https://issues.apache.org/jira/browse/CONNECTORS-39 Project: Lucene Connector Framework Issue Type: Improvement Components: Framework core Reporter: Karl Wright Assignee: Karl Wright The database abstraction layer in LCF does not permit someone to abstract from transaction management. That responsibility is delegated to a different class, which presumes that transaction management is not database-type dependent. Unfortunately, this is not the case. A better code structure would involve creating an abstract base class that performed the transaction management and caching, and causing all database implementations to be derived from it. Then, abstract methods for transaction begin and end could be readily defined. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CONNECTORS-35) Need a way to reset LCF when external conditions change
[ https://issues.apache.org/jira/browse/CONNECTORS-35?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-35. --- Resolution: Fixed Have decided that the current functionality is adequate, and no further work needs to be done. Need a way to reset LCF when external conditions change --- Key: CONNECTORS-35 URL: https://issues.apache.org/jira/browse/CONNECTORS-35 Project: Lucene Connector Framework Issue Type: Improvement Components: Framework agents process, Framework core, Framework crawler agent Reporter: Karl Wright Assignee: Karl Wright When a change is made external to LCF, such as a Solr configuration change, LCF needs some way for a user to signal that that change took place. For example, a button or link on the view output connection page might signal some undefined global change in the target of that output connection. A similar button or link on the repository connection view page might signal a corresponding change to the underlying repository. Clicking the button would do the following things: (1) It would clear the current version string for all documents that passed through that connection. This would guarantee that the documents would be reingested if and when they were processed the next time. (2) It would reset the last job time value for all jobs affected by the connection to zero. This would guarantee that all documents belonging to that job would be rechecked. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
