[jira] Updated: (CONNECTORS-91) Making the initialization commands more useable
[ https://issues.apache.org/jira/browse/CONNECTORS-91?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jettro Coenradie updated CONNECTORS-91: --- Attachment: commandsPatch.patch the patch with an improvement for the commands Making the initialization commands more useable --- Key: CONNECTORS-91 URL: https://issues.apache.org/jira/browse/CONNECTORS-91 Project: Lucene Connector Framework Issue Type: Improvement Components: Framework core Reporter: Jettro Coenradie Fix For: LCF Release 0.5 Attachments: commandsPatch.patch At the moment LCF comes with some classes that can be used to run command line to interact with the system. Examples are DBCreate, DBDrop and LockClean. I wanted to create a class that rebuilds my complete environment. So dropping a database, creating a database, cleaning the synch folder, registering agents, etc. Due to the structure of the classes with all the logic in the main method, I could not easily reuse these classes. In the patch I submit with issue I have refactored the current solution in a better reuseable solution that can still be called command line. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Connector architecture question and suggestion
Hi, I am having a look at the connectors. At the moment to my opinion the classes for (all) connectors are to big. This is partly due to the way the interfaces are structured and partly due to the implementation of html in java. For example the RssConnector now has almost 6000 lines of code and the JdbcConnector has almost 2000 lines of code. To my opinion this can be improved by making separate components for presenting and configuring the connectors in the crawler-ui web project and for the part needed by the runner. Abstracting the html from the actual classes can help a lot as well. Maybe some utility methods to make creating these html pages easier is nice as well. I am willing to investigate this path further, but I'd like to have ideas of other developers. It would be nice to know if others feel like this can be improved as well. It might be interesting to look at a technique like wicket for the ui part. Than you can package the html code together with the java code in one jar. No difficult repackaging is required and you can still create nice interfaces. I also read that others want to have a look at something like velocity, of course that can be a valid option as well. So what is your opinions about it? regards Jettro Coenradie http://www.gridshore.nl
[jira] Commented: (CONNECTORS-91) Making the initialization commands more useable
[ https://issues.apache.org/jira/browse/CONNECTORS-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898857#action_12898857 ] Jettro Coenradie commented on CONNECTORS-91: If you feel this is the way to go, I will change the other classes that have a main method as well. Making the initialization commands more useable --- Key: CONNECTORS-91 URL: https://issues.apache.org/jira/browse/CONNECTORS-91 Project: Lucene Connector Framework Issue Type: Improvement Components: Framework core Reporter: Jettro Coenradie Fix For: LCF Release 0.5 Attachments: commandsPatch.patch At the moment LCF comes with some classes that can be used to run command line to interact with the system. Examples are DBCreate, DBDrop and LockClean. I wanted to create a class that rebuilds my complete environment. So dropping a database, creating a database, cleaning the synch folder, registering agents, etc. Due to the structure of the classes with all the logic in the main method, I could not easily reuse these classes. In the patch I submit with issue I have refactored the current solution in a better reuseable solution that can still be called command line. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-19) Look into converting SOLR connector to use SolrJ java library
[ https://issues.apache.org/jira/browse/CONNECTORS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898859#action_12898859 ] Jettro Coenradie commented on CONNECTORS-19: We have a working solr connector that makes use of solr. This might be a good start. I might need to spend some time to make it run in the lcf build. We have a maven build to package it at the moment. If you are interested, let me know. Than I will spend the time on a patch. Look into converting SOLR connector to use SolrJ java library - Key: CONNECTORS-19 URL: https://issues.apache.org/jira/browse/CONNECTORS-19 Project: Lucene Connector Framework Issue Type: Improvement Components: Lucene/SOLR connector Reporter: Karl Wright Priority: Minor The SOLR connector currently uses its own multipart post code. It might be a good idea to convert it to use the SolrJ client api jar instead. This would require license confirmation, plus research to make sure there are no jar conflicts as a result, with any other connector. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CONNECTORS-19) Look into converting SOLR connector to use SolrJ java library
[ https://issues.apache.org/jira/browse/CONNECTORS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898859#action_12898859 ] Jettro Coenradie edited comment on CONNECTORS-19 at 8/16/10 7:45 AM: - We have a working solr connector that makes use of solrj. This might be a good start. I might need to spend some time to make it run in the lcf build. We have a maven build to package it at the moment. If you are interested, let me know. Than I will spend the time on a patch. was (Author: jettroc): We have a working solr connector that makes use of solr. This might be a good start. I might need to spend some time to make it run in the lcf build. We have a maven build to package it at the moment. If you are interested, let me know. Than I will spend the time on a patch. Look into converting SOLR connector to use SolrJ java library - Key: CONNECTORS-19 URL: https://issues.apache.org/jira/browse/CONNECTORS-19 Project: Lucene Connector Framework Issue Type: Improvement Components: Lucene/SOLR connector Reporter: Karl Wright Priority: Minor The SOLR connector currently uses its own multipart post code. It might be a good idea to convert it to use the SolrJ client api jar instead. This would require license confirmation, plus research to make sure there are no jar conflicts as a result, with any other connector. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
improving the build
Hi, I think the current download is pretty big. Is there is good reason that we do not use something like maven. Or at least, if you do not like maven, ivy to share dependencies between projects. Also enforces you to use libraries that are generally available. I would also love to have the (unit)tests closer to the actual code, hard to locate the tests at the moment Would like to hear the thoughts of the developers regards -- Jettro Coenradie http://www.gridshore.nl
[jira] Commented: (CONNECTORS-91) Making the initialization commands more useable
[ https://issues.apache.org/jira/browse/CONNECTORS-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898920#action_12898920 ] Karl Wright commented on CONNECTORS-91: --- It looks like this is simply using class-inheritance to separate out common functionality. As such, I'm in favor of including this contribution. Are there any subtleties I am missing? Making the initialization commands more useable --- Key: CONNECTORS-91 URL: https://issues.apache.org/jira/browse/CONNECTORS-91 Project: Lucene Connector Framework Issue Type: Improvement Components: Framework core Reporter: Jettro Coenradie Fix For: LCF Release 0.5 Attachments: commandsPatch.patch At the moment LCF comes with some classes that can be used to run command line to interact with the system. Examples are DBCreate, DBDrop and LockClean. I wanted to create a class that rebuilds my complete environment. So dropping a database, creating a database, cleaning the synch folder, registering agents, etc. Due to the structure of the classes with all the logic in the main method, I could not easily reuse these classes. In the patch I submit with issue I have refactored the current solution in a better reuseable solution that can still be called command line. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
RE: Connector architecture question and suggestion
I think that providing tools/help for implementing the UI pieces of connectors is a perfectly reasonable thing to do. However, I strongly believe that the UI components should remain described as part of the connector interfaces. Breaking their implementations out within individual connectors is also reasonable, but unless there is a compelling reason to refactor all the individual connectors in that way, I would hold off that project until there are better unit tests for the connector UI components. If you are willing to contribute such tests, I think going that way would be worth consideration. I'd like to see a more detailed proposal before I comment further. Thanks, Karl -Original Message- From: jettro.coenra...@gmail.com [mailto:jettro.coenra...@gmail.com] On Behalf Of ext Jettro Coenradie Sent: Monday, August 16, 2010 7:37 AM To: connectors-dev@incubator.apache.org Subject: Connector architecture question and suggestion Hi, I am having a look at the connectors. At the moment to my opinion the classes for (all) connectors are to big. This is partly due to the way the interfaces are structured and partly due to the implementation of html in java. For example the RssConnector now has almost 6000 lines of code and the JdbcConnector has almost 2000 lines of code. To my opinion this can be improved by making separate components for presenting and configuring the connectors in the crawler-ui web project and for the part needed by the runner. Abstracting the html from the actual classes can help a lot as well. Maybe some utility methods to make creating these html pages easier is nice as well. I am willing to investigate this path further, but I'd like to have ideas of other developers. It would be nice to know if others feel like this can be improved as well. It might be interesting to look at a technique like wicket for the ui part. Than you can package the html code together with the java code in one jar. No difficult repackaging is required and you can still create nice interfaces. I also read that others want to have a look at something like velocity, of course that can be a valid option as well. So what is your opinions about it? regards Jettro Coenradie http://www.gridshore.nl
RE: Suggestion for a Hippo repository connector
Connector contributions are always welcome, even if others are not using it as of yet. Can you create a ticket which describes the connector and its environment (including the security model the repository uses, and what authority connector is appropriate to use with it), and attach your connector as a patch? Thanks! Karl -Original Message- From: jettro.coenra...@gmail.com [mailto:jettro.coenra...@gmail.com] On Behalf Of ext Jettro Coenradie Sent: Monday, August 16, 2010 8:07 AM To: connectors-dev@incubator.apache.org Subject: Suggestion for a Hippo repository connector Hi, We have a basic hippo repository connection that we could make available for the LCF project. Are there people that would like to have such a connector? regards -- Jettro Coenradie http://www.gridshore.nl
[jira] Commented: (CONNECTORS-91) Making the initialization commands more useable
[ https://issues.apache.org/jira/browse/CONNECTORS-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898925#action_12898925 ] Jettro Coenradie commented on CONNECTORS-91: There should be no subtleties, I mainly moved code from the main method into a new method and indeed a bit of class-inheritance. Making the initialization commands more useable --- Key: CONNECTORS-91 URL: https://issues.apache.org/jira/browse/CONNECTORS-91 Project: Lucene Connector Framework Issue Type: Improvement Components: Framework core Reporter: Jettro Coenradie Fix For: LCF Release 0.5 Attachments: commandsPatch.patch At the moment LCF comes with some classes that can be used to run command line to interact with the system. Examples are DBCreate, DBDrop and LockClean. I wanted to create a class that rebuilds my complete environment. So dropping a database, creating a database, cleaning the synch folder, registering agents, etc. Due to the structure of the classes with all the logic in the main method, I could not easily reuse these classes. In the patch I submit with issue I have refactored the current solution in a better reuseable solution that can still be called command line. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Suggestion for a Hippo repository connector
I will do that, need to check for the libraries that Hippo needs, will create a report and come back with that. On Mon, Aug 16, 2010 at 2:52 PM, karl.wri...@nokia.com wrote: Connector contributions are always welcome, even if others are not using it as of yet. Can you create a ticket which describes the connector and its environment (including the security model the repository uses, and what authority connector is appropriate to use with it), and attach your connector as a patch? Thanks! Karl -Original Message- From: jettro.coenra...@gmail.com [mailto:jettro.coenra...@gmail.com] On Behalf Of ext Jettro Coenradie Sent: Monday, August 16, 2010 8:07 AM To: connectors-dev@incubator.apache.org Subject: Suggestion for a Hippo repository connector Hi, We have a basic hippo repository connection that we could make available for the LCF project. Are there people that would like to have such a connector? regards -- Jettro Coenradie http://www.gridshore.nl -- Jettro Coenradie http://www.gridshore.nl
Re: improving the build
We could use something like profiles in maven. That way you can easily configure which projects to compile and which not. That would include tests. I will have a look at it and come up with a proposal. On Mon, Aug 16, 2010 at 2:49 PM, karl.wri...@nokia.com wrote: Re maven: There is a wiki page describing the Maven dependencies; somebody needs to tackle this. If you want to volunteer, we'd love to hear your proposal. However, please remember that you really must be sure to retain the connector conditional compilation structure as is currently in place, for license reasons. Re unit tests: The Junit test code was actually placed carefully based on the above considerations. In other words, you can't run a test that requires connectors x,y,z unless those connectors have actually been built. Similarly, you may think in terms of testing a specific connector by including tests for that connector, but those tests cannot use any OTHER connectors or you will break the build, which is why any tests that use multiple connectors must be at the module level. Karl -Original Message- From: jettro.coenra...@gmail.com [mailto:jettro.coenra...@gmail.com] On Behalf Of ext Jettro Coenradie Sent: Monday, August 16, 2010 8:21 AM To: connectors-dev@incubator.apache.org Subject: improving the build Hi, I think the current download is pretty big. Is there is good reason that we do not use something like maven. Or at least, if you do not like maven, ivy to share dependencies between projects. Also enforces you to use libraries that are generally available. I would also love to have the (unit)tests closer to the actual code, hard to locate the tests at the moment Would like to hear the thoughts of the developers regards -- Jettro Coenradie http://www.gridshore.nl -- Jettro Coenradie http://www.gridshore.nl
[jira] Commented: (CONNECTORS-19) Look into converting SOLR connector to use SolrJ java library
[ https://issues.apache.org/jira/browse/CONNECTORS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898928#action_12898928 ] Jettro Coenradie commented on CONNECTORS-19: I will have a good look at the dependencies and the functionality. If satisfied, I will supply a patch that other can check as well. Look into converting SOLR connector to use SolrJ java library - Key: CONNECTORS-19 URL: https://issues.apache.org/jira/browse/CONNECTORS-19 Project: Lucene Connector Framework Issue Type: Improvement Components: Lucene/SOLR connector Reporter: Karl Wright Priority: Minor The SOLR connector currently uses its own multipart post code. It might be a good idea to convert it to use the SolrJ client api jar instead. This would require license confirmation, plus research to make sure there are no jar conflicts as a result, with any other connector. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Connector architecture question and suggestion
Oke, I will come up with a proposal for breaking up the components in a way that still enables us to easily keep the current adapters in their own structure. As for the UI testen, it is always a beast. We do have some experience with tools like Webdriver and others. Will have a look at that as well. On Mon, Aug 16, 2010 at 2:41 PM, karl.wri...@nokia.com wrote: I think that providing tools/help for implementing the UI pieces of connectors is a perfectly reasonable thing to do. However, I strongly believe that the UI components should remain described as part of the connector interfaces. Breaking their implementations out within individual connectors is also reasonable, but unless there is a compelling reason to refactor all the individual connectors in that way, I would hold off that project until there are better unit tests for the connector UI components. If you are willing to contribute such tests, I think going that way would be worth consideration. I'd like to see a more detailed proposal before I comment further. Thanks, Karl -Original Message- From: jettro.coenra...@gmail.com [mailto:jettro.coenra...@gmail.com] On Behalf Of ext Jettro Coenradie Sent: Monday, August 16, 2010 7:37 AM To: connectors-dev@incubator.apache.org Subject: Connector architecture question and suggestion Hi, I am having a look at the connectors. At the moment to my opinion the classes for (all) connectors are to big. This is partly due to the way the interfaces are structured and partly due to the implementation of html in java. For example the RssConnector now has almost 6000 lines of code and the JdbcConnector has almost 2000 lines of code. To my opinion this can be improved by making separate components for presenting and configuring the connectors in the crawler-ui web project and for the part needed by the runner. Abstracting the html from the actual classes can help a lot as well. Maybe some utility methods to make creating these html pages easier is nice as well. I am willing to investigate this path further, but I'd like to have ideas of other developers. It would be nice to know if others feel like this can be improved as well. It might be interesting to look at a technique like wicket for the ui part. Than you can package the html code together with the java code in one jar. No difficult repackaging is required and you can still create nice interfaces. I also read that others want to have a look at something like velocity, of course that can be a valid option as well. So what is your opinions about it? regards Jettro Coenradie http://www.gridshore.nl -- Jettro Coenradie http://www.gridshore.nl
Re: Project status and name
I think it needs to be brought up on the gene...@incubator.a.o list. I'm not sure it's ever happened before. FWIW: I'm +1 on ACF. On Aug 15, 2010, at 5:16 PM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: Grant, what are the next steps here? Karl -Original Message- From: Wright Karl (Nokia-MS/Cambridge) Sent: Friday, August 13, 2010 5:53 PM To: connectors-dev@incubator.apache.org Subject: RE: Project status and name I don't think the name change is tied at all to the incubation status. Are we ready to call a vote? After much consideration, +1 for ACF. Karl From: ext Jack Krupansky [jack.krupan...@lucidimagination.com] Sent: Friday, August 13, 2010 5:51 PM To: connectors-dev@incubator.apache.org Subject: Re: Project status and name Any consensus on the name change? I am okay with either name. ACF should be fine. Presumably the nominal name change is contingent on its project status as no longer incubating under Lucene? -- Jack Krupansky -- From: Grant Ingersoll gsing...@apache.org Sent: Tuesday, August 10, 2010 5:19 PM To: connectors-dev@incubator.apache.org Subject: Re: Project status and name I'd leave it open for another day or two. -Grant On Aug 10, 2010, at 2:16 PM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: Shall we call a vote on the name change? Or should we leave the floor open for other proposals for a while? Karl -Original Message- From: ext Grant Ingersoll [mailto:gsing...@apache.org] Sent: Tuesday, August 10, 2010 2:09 PM To: connectors-dev@incubator.apache.org Subject: Re: Project status and name On Aug 10, 2010, at 8:05 AM, karl.wri...@nokia.com karl.wri...@nokia.com wrote: Folks, Lucene Connectors Framework is currently an incubating subproject of Lucene. The PMC has indicated that it's not thrilled with the idea of LCF being a subproject, Minor clarification: The PMC hasn't said no at this point, but it also hasn't been discussed. Given some of the recent restructuring, I was merely speculating privately to Karl that it likely would not accept it, but that is not anything official. Not that it needs to be decided now anyway. FWIW, the Board isn't usually happy w/ PMC's that are umbrella projects, with separate SVN, JIRA, etc. See the discussions in the archives around Mahout, Nutch, Lucy and Tika. When LCF was brought into incubation, there wasn't as much of a concern as there is now, so it is not that LCF did anything wrong. Besides, LCF is really independent of Lucene and useful w/o connecting to search and should have it's own management anyway. and that its status should change at some point in the future. Note that this status change would be theoretically independent of the project name, but potentially we'd consider changing the project name at that time as well. There's beginning to be a considerable amount of content floating around that talks about LCF. If there is a possibility of a name change for this project, I'd like to open the discussion as to whether we should change the name, and if so, what to. FWIW, the only other possibility I've heard mentioned so far is Apache Connectors Framework. I think this works well and abbreviates nicely to ACF (of course, I was the one who suggested it, so I'm biased). Note, there is no reason it can't be called the Lucene Connectors Framework, but that might pigeonhole it such that people think it only works with Lucene, which simply isn't true. I agree, we should do this change sooner rather than later, if it is going to be done. -Grant -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem docs using Solr/Lucene: http://www.lucidimagination.com/search -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search
