[jira] Updated: (CONNECTORS-92) Move from ant to maven or other build system with decent library management
[ https://issues.apache.org/jira/browse/CONNECTORS-92?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jettro Coenradie updated CONNECTORS-92: --- Attachment: patch-connectors.zip Not a patch, but a zip containing a few poms for the connectors. The null connector and the jdbc connector have a pom Move from ant to maven or other build system with decent library management --- Key: CONNECTORS-92 URL: https://issues.apache.org/jira/browse/CONNECTORS-92 Project: Apache Connectors Framework Issue Type: Wish Components: Build Reporter: Jettro Coenradie Assignee: Karl Wright Attachments: maven-poms-including-start-jar.patch, maven-poms-problem-starting-jetty-and-derby.patch, maven-start-jar.patch, move-to-maven-acf-framework.patch, patch-connectors.zip, Screen shot 2010-08-23 at 16.31.07.png I am looking at the current project structure. If we want to make another build tool available I think we need to change the directory structure. I tried to place a suggestion in an image. Can you please have a look at it. If we agree that this is a good way to go, than I will continue to work on a patch. Which might be a bit hard with all these changing directories, but I'll do my best to at least get an idea whether it would be working. So I have three questions: - Do you want to move to maven or put maven next to ant? - Do you prefer another build mechanism [ant with ivy, gradle, maven3] - Do you have an idea about the amount of scripts that need to be changed if we change the project structure The image of a possible project layout (that is based on the maven standards) is attached to the issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-92) Move from ant to maven or other build system with decent library management
[ https://issues.apache.org/jira/browse/CONNECTORS-92?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12907908#action_12907908 ] Jettro Coenradie commented on CONNECTORS-92: I think maven is not the right tool at the moment due to the libraries that are used that are not available in any repository, the way the sample is started. The dependencies that seem to be copied to each project. I am spending a lot of time on creating an assembly, but that is not really the part where maven shines. Move from ant to maven or other build system with decent library management --- Key: CONNECTORS-92 URL: https://issues.apache.org/jira/browse/CONNECTORS-92 Project: Apache Connectors Framework Issue Type: Wish Components: Build Reporter: Jettro Coenradie Assignee: Karl Wright Attachments: maven-poms-including-start-jar.patch, maven-poms-problem-starting-jetty-and-derby.patch, maven-start-jar.patch, move-to-maven-acf-framework.patch, patch-connectors.zip, Screen shot 2010-08-23 at 16.31.07.png I am looking at the current project structure. If we want to make another build tool available I think we need to change the directory structure. I tried to place a suggestion in an image. Can you please have a look at it. If we agree that this is a good way to go, than I will continue to work on a patch. Which might be a bit hard with all these changing directories, but I'll do my best to at least get an idea whether it would be working. So I have three questions: - Do you want to move to maven or put maven next to ant? - Do you prefer another build mechanism [ant with ivy, gradle, maven3] - Do you have an idea about the amount of scripts that need to be changed if we change the project structure The image of a possible project layout (that is based on the maven standards) is attached to the issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-92) Move from ant to maven or other build system with decent library management
[ https://issues.apache.org/jira/browse/CONNECTORS-92?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12907358#action_12907358 ] Jettro Coenradie commented on CONNECTORS-92: I worked on it tonight but I decided to stop. This path is not leading in a direction that I would like. To make most out of maven I would like to change more than you would be willing to right now. I cannot blame you, because you have something working right now. Maybe someone else wants to step in and finish what I have done. I can submit another patch with the stuff I have right now. Move from ant to maven or other build system with decent library management --- Key: CONNECTORS-92 URL: https://issues.apache.org/jira/browse/CONNECTORS-92 Project: Apache Connectors Framework Issue Type: Wish Components: Build Reporter: Jettro Coenradie Assignee: Karl Wright Attachments: maven-poms-including-start-jar.patch, maven-poms-problem-starting-jetty-and-derby.patch, move-to-maven-acf-framework.patch, Screen shot 2010-08-23 at 16.31.07.png I am looking at the current project structure. If we want to make another build tool available I think we need to change the directory structure. I tried to place a suggestion in an image. Can you please have a look at it. If we agree that this is a good way to go, than I will continue to work on a patch. Which might be a bit hard with all these changing directories, but I'll do my best to at least get an idea whether it would be working. So I have three questions: - Do you want to move to maven or put maven next to ant? - Do you prefer another build mechanism [ant with ivy, gradle, maven3] - Do you have an idea about the amount of scripts that need to be changed if we change the project structure The image of a possible project layout (that is based on the maven standards) is attached to the issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: [VOTE] Pick your preferred name
Apache Yukon Apache Macon Apache Manifold On Wed, Sep 1, 2010 at 9:42 AM, Simon Willnauer simon.willna...@googlemail.com wrote: Apache Manifold Apache Connectors Framework Apache Omni On Wed, Sep 1, 2010 at 2:21 AM, Karl Wright daddy...@gmail.com wrote: I think this vote will conclusively demonstrate the community position to the solons at incubator. They can choose to ignore the vote, of course, but they won't be able to say we didn't consider alternatives. ;-) Karl On Tue, Aug 31, 2010 at 8:16 PM, Mark Miller markrmil...@gmail.com wrote: I think its pretty clear that unless the lords of the incubator *force* us to do otherwise, we should just leave it as Apache Connectors Framework. The vote is going to end up reflecting that is my bet anyway. I havn't seen anyone that's actually involved (if you can say anyone beyond karl is really involved anyway) is really dying for a change. - Mark On 8/31/10 8:07 PM, Robert Muir wrote: you can ignore my vote really :) but i was thinking from a practical perspective, this one involves the least code changes. plus, with existing projects named things like apache DB and apache http server, i don't understand the fuss. On Tue, Aug 31, 2010 at 8:03 PM, Karl Wright daddy...@gmail.com wrote: Well, you've made your feelings clear. ;-) Karl On Tue, Aug 31, 2010 at 7:58 PM, Robert Muir rcm...@gmail.com wrote: Apache Connectors Framework Apache Connectors Framework Apache Connectors Framework (sorry) On Tue, Aug 31, 2010 at 6:44 PM, Karl Wright daddy...@gmail.com wrote: I know this is un-Apache-like, but please respond to the following list with a selection, in order, of the top three names for the project currently known as Apache Connectors Framework. The choices are: Apache Connectors Framework Apache Acromantula Apache Manifold Apache ManifoldCF Apache Multiplex Apache Lucon Apache Lukon Apache Yukon Apache Macon Apache Omni Apache Omnivore Apache CMCF (yes, I just invented that one ;-) ) Apache Multivore (yes, I just invented that one too. ;-) ) I don't think I missed any? If I did, chastise me severely please. ;-) Karl -- Robert Muir rcm...@gmail.com -- Jettro Coenradie http://www.gridshore.nl
[jira] Commented: (CONNECTORS-92) Move from ant to maven or other build system with decent library management
[ https://issues.apache.org/jira/browse/CONNECTORS-92?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12904561#action_12904561 ] Jettro Coenradie commented on CONNECTORS-92: As for the start.jar I do not see a problem. I think I am almost there. THe classpath already contains the lib part, so I only need to add the dependencies to the jetty runner project. As for the example code, I do not mind to keep using and to create the example. I only wanted to have maven to make it easier to setup my development environment and to do the dependency management. I'll try to come up with an improve pom for the start.jar, if that is not enough please let me know. Move from ant to maven or other build system with decent library management --- Key: CONNECTORS-92 URL: https://issues.apache.org/jira/browse/CONNECTORS-92 Project: Apache Connectors Framework Issue Type: Wish Components: Build Reporter: Jettro Coenradie Assignee: Karl Wright Attachments: maven-poms-including-start-jar.patch, maven-poms-problem-starting-jetty-and-derby.patch, move-to-maven-acf-framework.patch, Screen shot 2010-08-23 at 16.31.07.png I am looking at the current project structure. If we want to make another build tool available I think we need to change the directory structure. I tried to place a suggestion in an image. Can you please have a look at it. If we agree that this is a good way to go, than I will continue to work on a patch. Which might be a bit hard with all these changing directories, but I'll do my best to at least get an idea whether it would be working. So I have three questions: - Do you want to move to maven or put maven next to ant? - Do you prefer another build mechanism [ant with ivy, gradle, maven3] - Do you have an idea about the amount of scripts that need to be changed if we change the project structure The image of a possible project layout (that is based on the maven standards) is attached to the issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (CONNECTORS-92) Move from ant to maven or other build system with decent library management
[ https://issues.apache.org/jira/browse/CONNECTORS-92?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jettro Coenradie updated CONNECTORS-92: --- Attachment: maven-poms-problem-starting-jetty-and-derby.patch This is a patch that addes poms to the framework part. It runs after you have installed the one missing jar. In the end the mvn clean install should be successful in the root of the framework directory In the crawler-ui project you can do : mvn clean jetty:run-war This will start up the crawler-ui, only after you have copied the conenctors that you need to the src/config/local/connector-lib Than asking for a list of connectors results in a database error. I just cannot get the Derby instance to run. The database is created but if seems not to be running. Any help in this area would be appreciated. Caused by: ERROR 42Y07: Schema 'ACF' does not exist at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(Unknown Source) at org.apache.derby.impl.sql.compile.QueryTreeNode.getSchemaDescriptor(Unknown Source) at org.apache.derby.impl.sql.compile.QueryTreeNode.getSchemaDescriptor(Unknown Source) at org.apache.derby.impl.sql.compile.FromBaseTable.bindTableDescriptor(Unknown Source) at org.apache.derby.impl.sql.compile.FromBaseTable.bindNonVTITables(Unknown Source) at org.apache.derby.impl.sql.compile.FromList.bindTables(Unknown Source) at org.apache.derby.impl.sql.compile.SelectNode.bindNonVTITables(Unknown Source) at org.apache.derby.impl.sql.compile.DMLStatementNode.bindTables(Unknown Source) at org.apache.derby.impl.sql.compile.DMLStatementNode.bind(Unknown Source) at org.apache.derby.impl.sql.compile.CursorNode.bindStatement(Unknown Source) at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source) at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source) ... 4 more Move from ant to maven or other build system with decent library management --- Key: CONNECTORS-92 URL: https://issues.apache.org/jira/browse/CONNECTORS-92 Project: Apache Connectors Framework Issue Type: Wish Components: Build Reporter: Jettro Coenradie Attachments: maven-poms-problem-starting-jetty-and-derby.patch, move-to-maven-acf-framework.patch, Screen shot 2010-08-23 at 16.31.07.png I am looking at the current project structure. If we want to make another build tool available I think we need to change the directory structure. I tried to place a suggestion in an image. Can you please have a look at it. If we agree that this is a good way to go, than I will continue to work on a patch. Which might be a bit hard with all these changing directories, but I'll do my best to at least get an idea whether it would be working. So I have three questions: - Do you want to move to maven or put maven next to ant? - Do you prefer another build mechanism [ant with ivy, gradle, maven3] - Do you have an idea about the amount of scripts that need to be changed if we change the project structure The image of a possible project layout (that is based on the maven standards) is attached to the issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-92) Move from ant to maven or other build system with decent library management
[ https://issues.apache.org/jira/browse/CONNECTORS-92?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12903476#action_12903476 ] Jettro Coenradie commented on CONNECTORS-92: Oke, thats fine. But is the projects *-servlet a war? is that the web project? Move from ant to maven or other build system with decent library management --- Key: CONNECTORS-92 URL: https://issues.apache.org/jira/browse/CONNECTORS-92 Project: Apache Connectors Framework Issue Type: Wish Components: Build Reporter: Jettro Coenradie Attachments: move-to-maven-acf-framework.patch, Screen shot 2010-08-23 at 16.31.07.png I am looking at the current project structure. If we want to make another build tool available I think we need to change the directory structure. I tried to place a suggestion in an image. Can you please have a look at it. If we agree that this is a good way to go, than I will continue to work on a patch. Which might be a bit hard with all these changing directories, but I'll do my best to at least get an idea whether it would be working. So I have three questions: - Do you want to move to maven or put maven next to ant? - Do you prefer another build mechanism [ant with ivy, gradle, maven3] - Do you have an idea about the amount of scripts that need to be changed if we change the project structure The image of a possible project layout (that is based on the maven standards) is attached to the issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (CONNECTORS-92) Move from ant to maven or other build system with decent library management
[ https://issues.apache.org/jira/browse/CONNECTORS-92?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jettro Coenradie updated CONNECTORS-92: --- Attachment: move-to-maven-acf-framework.patch This is a first go on moving to maven. I started refactoring the framework part. This should give a good idea of what is possible. Of course we are nog there. I did not really touch any of the scripts. There is also one dependency you have to install locally. You'll see it when you try to run maven. Advice on ho to install it locally is provided when it fails. I am not sure if this patch will run, it is a big one. Hope for the best When the patch is successfully applied, go to the modules/framework folder and try: mvn clean install then step into crawler-ui and do mvn clean jetty:run-war Use you browser to go to localhost:8080 and you can see the crawler-ui web app. If you connect to an existing database, you should be able to see the configure connections lists. Not more than that, than we need to put the connectors on the classpath. I'd be happy to do something similar for the connectors. But than we must be sure that this is the way to go. It takes a lot of time. Move from ant to maven or other build system with decent library management --- Key: CONNECTORS-92 URL: https://issues.apache.org/jira/browse/CONNECTORS-92 Project: Apache Connectors Framework Issue Type: Wish Components: Build Reporter: Jettro Coenradie Attachments: move-to-maven-acf-framework.patch, Screen shot 2010-08-23 at 16.31.07.png I am looking at the current project structure. If we want to make another build tool available I think we need to change the directory structure. I tried to place a suggestion in an image. Can you please have a look at it. If we agree that this is a good way to go, than I will continue to work on a patch. Which might be a bit hard with all these changing directories, but I'll do my best to at least get an idea whether it would be working. So I have three questions: - Do you want to move to maven or put maven next to ant? - Do you prefer another build mechanism [ant with ivy, gradle, maven3] - Do you have an idea about the amount of scripts that need to be changed if we change the project structure The image of a possible project layout (that is based on the maven standards) is attached to the issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-92) Move from ant to maven or other build system with decent library management
[ https://issues.apache.org/jira/browse/CONNECTORS-92?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12902834#action_12902834 ] Jettro Coenradie commented on CONNECTORS-92: Very cool Move from ant to maven or other build system with decent library management --- Key: CONNECTORS-92 URL: https://issues.apache.org/jira/browse/CONNECTORS-92 Project: Apache Connectors Framework Issue Type: Wish Components: Build Reporter: Jettro Coenradie Attachments: move-to-maven-acf-framework.patch, Screen shot 2010-08-23 at 16.31.07.png I am looking at the current project structure. If we want to make another build tool available I think we need to change the directory structure. I tried to place a suggestion in an image. Can you please have a look at it. If we agree that this is a good way to go, than I will continue to work on a patch. Which might be a bit hard with all these changing directories, but I'll do my best to at least get an idea whether it would be working. So I have three questions: - Do you want to move to maven or put maven next to ant? - Do you prefer another build mechanism [ant with ivy, gradle, maven3] - Do you have an idea about the amount of scripts that need to be changed if we change the project structure The image of a possible project layout (that is based on the maven standards) is attached to the issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (CONNECTORS-91) Making the initialization commands more useable
[ https://issues.apache.org/jira/browse/CONNECTORS-91?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jettro Coenradie updated CONNECTORS-91: --- Attachment: change_commands.patch Some strange things are happening, not sure what went wrong. I did do an svn up, I am sure of that. Nevertheless, I think I have it working now. You might need to change the depth of which to apply the patch. I recreated the patch with intellij and it uses one folder of my own. The following command strips of this folder patch -p1 -i ~/change_commands.patch I tried it on a clean checkout of the project locally and it seems to work Hope it works now, sorry I did not try it myself before Making the initialization commands more useable --- Key: CONNECTORS-91 URL: https://issues.apache.org/jira/browse/CONNECTORS-91 Project: Apache Connectors Framework Issue Type: Improvement Components: Framework core Reporter: Jettro Coenradie Fix For: LCF Release 0.5 Attachments: change_commands.patch At the moment LCF comes with some classes that can be used to run command line to interact with the system. Examples are DBCreate, DBDrop and LockClean. I wanted to create a class that rebuilds my complete environment. So dropping a database, creating a database, cleaning the synch folder, registering agents, etc. Due to the structure of the classes with all the logic in the main method, I could not easily reuse these classes. In the patch I submit with issue I have refactored the current solution in a better reuseable solution that can still be called command line. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (CONNECTORS-91) Making the initialization commands more useable
[ https://issues.apache.org/jira/browse/CONNECTORS-91?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jettro Coenradie updated CONNECTORS-91: --- Attachment: (was: changesToCommandClasses.patch) Making the initialization commands more useable --- Key: CONNECTORS-91 URL: https://issues.apache.org/jira/browse/CONNECTORS-91 Project: Apache Connectors Framework Issue Type: Improvement Components: Framework core Reporter: Jettro Coenradie Fix For: LCF Release 0.5 Attachments: change_commands.patch At the moment LCF comes with some classes that can be used to run command line to interact with the system. Examples are DBCreate, DBDrop and LockClean. I wanted to create a class that rebuilds my complete environment. So dropping a database, creating a database, cleaning the synch folder, registering agents, etc. Due to the structure of the classes with all the logic in the main method, I could not easily reuse these classes. In the patch I submit with issue I have refactored the current solution in a better reuseable solution that can still be called command line. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (CONNECTORS-91) Making the initialization commands more useable
[ https://issues.apache.org/jira/browse/CONNECTORS-91?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jettro Coenradie updated CONNECTORS-91: --- Comment: was deleted (was: sorry, pushed the wrong button I guess) Making the initialization commands more useable --- Key: CONNECTORS-91 URL: https://issues.apache.org/jira/browse/CONNECTORS-91 Project: Apache Connectors Framework Issue Type: Improvement Components: Framework core Reporter: Jettro Coenradie Fix For: LCF Release 0.5 Attachments: change_commands.patch At the moment LCF comes with some classes that can be used to run command line to interact with the system. Examples are DBCreate, DBDrop and LockClean. I wanted to create a class that rebuilds my complete environment. So dropping a database, creating a database, cleaning the synch folder, registering agents, etc. Due to the structure of the classes with all the logic in the main method, I could not easily reuse these classes. In the patch I submit with issue I have refactored the current solution in a better reuseable solution that can still be called command line. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (CONNECTORS-91) Making the initialization commands more useable
[ https://issues.apache.org/jira/browse/CONNECTORS-91?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jettro Coenradie updated CONNECTORS-91: --- Comment: was deleted (was: Some strange things are happening, not sure what went wrong. I did do an svn up, I am sure of that. Nevertheless, I think I have it working now. You might need to change the depth of which to apply the patch. I recreated the patch with intellij and it uses one folder of my own. The following command strips of this folder patch -p1 -i ~/change_commands.patch I tried it on a clean checkout of the project locally and it seems to work Hope it works now, sorry I did not try it myself before) Making the initialization commands more useable --- Key: CONNECTORS-91 URL: https://issues.apache.org/jira/browse/CONNECTORS-91 Project: Apache Connectors Framework Issue Type: Improvement Components: Framework core Reporter: Jettro Coenradie Fix For: LCF Release 0.5 Attachments: change_commands.patch At the moment LCF comes with some classes that can be used to run command line to interact with the system. Examples are DBCreate, DBDrop and LockClean. I wanted to create a class that rebuilds my complete environment. So dropping a database, creating a database, cleaning the synch folder, registering agents, etc. Due to the structure of the classes with all the logic in the main method, I could not easily reuse these classes. In the patch I submit with issue I have refactored the current solution in a better reuseable solution that can still be called command line. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Need an opinion, on whether to change package or not
+1 for a complete change On Mon, Aug 23, 2010 at 6:34 AM, Mark Miller markrmil...@gmail.com wrote: +1 to renaming the package - nows the time. - Mark http://www.lucidimagination.com (mobile) On Aug 22, 2010, at 8:01 PM, Jack Krupansky jack.krupan...@lucidimagination.com wrote: +1 -- Jack Krupansky -- From: Karl Wright daddy...@gmail.com Sent: Sunday, August 22, 2010 1:49 PM To: connectors-dev connectors-dev@incubator.apache.org Subject: Need an opinion, on whether to change package or not Consider this an official request for a vote. +1 indicates you think we should change the following in the source code, as soon as is practical: org.apache.lcf.xxx - org.apache.acf.xxx All classes LCF.java and LCFException.java should change to ACF.java and ACFException.java Bear in mind that users of ACF/LCF who currently have existing database instances will need to reinitialize those instances if we do this change. This is because the class names of connectors are stored in the database when the connector is registered. (FWIW, my vote on this is -1. It doesn't seem worth the disruption. But I will of course abide by the consensus.) Vote will be considered closed by Wednesday evening, so vote early (and often. ;-)) Karl -- Jettro Coenradie http://www.gridshore.nl
[jira] Commented: (CONNECTORS-91) Making the initialization commands more useable
[ https://issues.apache.org/jira/browse/CONNECTORS-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12901316#action_12901316 ] Jettro Coenradie commented on CONNECTORS-91: Hmm, I think the logging option is better, if people provide the right configuration you have what you need and even more. But I understand what you mean with the main method implementation. I'll add it back and provide a new patch. I also tried the sample with the new classes and it all seems to work. Is that good enough? Making the initialization commands more useable --- Key: CONNECTORS-91 URL: https://issues.apache.org/jira/browse/CONNECTORS-91 Project: Apache Connectors Framework Issue Type: Improvement Components: Framework core Reporter: Jettro Coenradie Fix For: LCF Release 0.5 Attachments: change_commands.patch At the moment LCF comes with some classes that can be used to run command line to interact with the system. Examples are DBCreate, DBDrop and LockClean. I wanted to create a class that rebuilds my complete environment. So dropping a database, creating a database, cleaning the synch folder, registering agents, etc. Due to the structure of the classes with all the logic in the main method, I could not easily reuse these classes. In the patch I submit with issue I have refactored the current solution in a better reuseable solution that can still be called command line. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (CONNECTORS-91) Making the initialization commands more useable
[ https://issues.apache.org/jira/browse/CONNECTORS-91?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jettro Coenradie updated CONNECTORS-91: --- Attachment: change_commands_with_system_err_println.patch added system err println lines back to the main methods Making the initialization commands more useable --- Key: CONNECTORS-91 URL: https://issues.apache.org/jira/browse/CONNECTORS-91 Project: Apache Connectors Framework Issue Type: Improvement Components: Framework core Reporter: Jettro Coenradie Fix For: LCF Release 0.5 Attachments: change_commands.patch, change_commands_with_system_err_println.patch At the moment LCF comes with some classes that can be used to run command line to interact with the system. Examples are DBCreate, DBDrop and LockClean. I wanted to create a class that rebuilds my complete environment. So dropping a database, creating a database, cleaning the synch folder, registering agents, etc. Due to the structure of the classes with all the logic in the main method, I could not easily reuse these classes. In the patch I submit with issue I have refactored the current solution in a better reuseable solution that can still be called command line. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (CONNECTORS-91) Making the initialization commands more useable
[ https://issues.apache.org/jira/browse/CONNECTORS-91?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jettro Coenradie updated CONNECTORS-91: --- Attachment: change_commands_with_system_err_println_v2.patch Sorry about the errors, I was a little bit to quick. I double checked all locations of printing the messages and the messages themselves. They should all be fine now. Making the initialization commands more useable --- Key: CONNECTORS-91 URL: https://issues.apache.org/jira/browse/CONNECTORS-91 Project: Apache Connectors Framework Issue Type: Improvement Components: Framework core Reporter: Jettro Coenradie Fix For: LCF Release 0.5 Attachments: change_commands.patch, change_commands_with_system_err_println.patch, change_commands_with_system_err_println_v2.patch At the moment LCF comes with some classes that can be used to run command line to interact with the system. Examples are DBCreate, DBDrop and LockClean. I wanted to create a class that rebuilds my complete environment. So dropping a database, creating a database, cleaning the synch folder, registering agents, etc. Due to the structure of the classes with all the logic in the main method, I could not easily reuse these classes. In the patch I submit with issue I have refactored the current solution in a better reuseable solution that can still be called command line. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Question about the json library
I know maven is another issue, but it is nice if the version is available through a maven repository. Than other build tools can find it as well. It is available for download through: http://mvnrepository.com/artifact/org.json/json or from the central maven repo: http://repo2.maven.org/maven2/org/json/json/20090211/json-20090211.jar Jettro On Mon, Aug 23, 2010 at 4:16 PM, karl.wri...@nokia.com wrote: The sources were downloaded from www.json.org, and are licensed accordingly. There is no build available from www.json.org. If you know of a prebuilt version of these sources, by all means point us at it. Mavenization is a different issue, and will have to be done independently. Karl -Original Message- From: jettro.coenra...@gmail.com [mailto:jettro.coenra...@gmail.com] On Behalf Of ext Jettro Coenradie Sent: Monday, August 23, 2010 10:04 AM To: connectors-dev@incubator.apache.org Subject: Question about the json library I am looking at the classes that come with the current trunk checkout and I see that a custom jar of json is created. Can someone explain why this is? Could we also take one from a maven repository? thanks, Jettro -- Jettro Coenradie http://www.gridshore.nl -- Jettro Coenradie http://www.gridshore.nl
[jira] Updated: (CONNECTORS-92) Move from ant to maven or other build system with decent library management
[ https://issues.apache.org/jira/browse/CONNECTORS-92?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jettro Coenradie updated CONNECTORS-92: --- Attachment: Screen shot 2010-08-23 at 16.31.07.png idea of the directory structure Move from ant to maven or other build system with decent library management --- Key: CONNECTORS-92 URL: https://issues.apache.org/jira/browse/CONNECTORS-92 Project: Apache Connectors Framework Issue Type: Wish Components: Build Reporter: Jettro Coenradie Attachments: Screen shot 2010-08-23 at 16.31.07.png I am looking at the current project structure. If we want to make another build tool available I think we need to change the directory structure. I tried to place a suggestion in an image. Can you please have a look at it. If we agree that this is a good way to go, than I will continue to work on a patch. Which might be a bit hard with all these changing directories, but I'll do my best to at least get an idea whether it would be working. So I have three questions: - Do you want to move to maven or put maven next to ant? - Do you prefer another build mechanism [ant with ivy, gradle, maven3] - Do you have an idea about the amount of scripts that need to be changed if we change the project structure The image of a possible project layout (that is based on the maven standards) is attached to the issue -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: improving the build
That is a good idea, maven can call ant to execute tasks. The jars are available in the maven repository and should therefore not be to hard to make available to the ant build. Would be nice to have an idea of the amount of scripts that we need to alter to make this happen. I also see a lot of shell scripts that might need attention if we change this. - Jettro On Mon, Aug 23, 2010 at 4:52 PM, karl.wri...@nokia.com wrote: Re: build preferences Continuing to have an ant build is actually pretty important for some modes of delivery. I'm specifically thinking of debian and Ubuntu packaging here. Maven does not work well with these packaging schemes because it's too all-encompassing. We therefore need a way of doing builds locally, without pulling things down from a mirror. My original thought was that we'd have multiple layers - ant being the most basic, with a maven wrapper available to pull down what the ant build needed, and have the maven build call ant underneath. I don't know how realistic that is, but it does solve all the problems if it can be done that way. Karl From: jettro.coenra...@gmail.com [mailto:jettro.coenra...@gmail.com] On Behalf Of ext Jettro Coenradie Sent: Monday, August 23, 2010 10:43 AM To: connectors-dev@incubator.apache.org Subject: Re: improving the build I am looking at the current project structure. If we want to make another build tool available I think we need to change the directory structure. I tried to place a suggestion in an image. Can you please have a look at it. If we agree that this is a good way to go, than I will continue to work on a patch. Which might be a bit hard with all these changing directories, but I'll do my best to at least get an idea whether it would be working. So I have three questions: - Do you want to move to maven or put maven next to ant? - Do you prefer another build mechanism [ant with ivy, gradle, maven3] - Do you have an idea about the amount of scripts that need to be changed if we change the project structure The image of a possible project layout (that is based on the maven standards) is attached to the mail Jettro On Mon, Aug 16, 2010 at 3:12 PM, Jettro Coenradie jettro.coenra...@gridshore.nlmailto:jettro.coenra...@gridshore.nl wrote: We could use something like profiles in maven. That way you can easily configure which projects to compile and which not. That would include tests. I will have a look at it and come up with a proposal. On Mon, Aug 16, 2010 at 2:49 PM, karl.wri...@nokia.commailto: karl.wri...@nokia.com wrote: Re maven: There is a wiki page describing the Maven dependencies; somebody needs to tackle this. If you want to volunteer, we'd love to hear your proposal. However, please remember that you really must be sure to retain the connector conditional compilation structure as is currently in place, for license reasons. Re unit tests: The Junit test code was actually placed carefully based on the above considerations. In other words, you can't run a test that requires connectors x,y,z unless those connectors have actually been built. Similarly, you may think in terms of testing a specific connector by including tests for that connector, but those tests cannot use any OTHER connectors or you will break the build, which is why any tests that use multiple connectors must be at the module level. Karl -Original Message- From: jettro.coenra...@gmail.commailto:jettro.coenra...@gmail.com [mailto:jettro.coenra...@gmail.commailto:jettro.coenra...@gmail.com] On Behalf Of ext Jettro Coenradie Sent: Monday, August 16, 2010 8:21 AM To: connectors-dev@incubator.apache.orgmailto: connectors-dev@incubator.apache.org Subject: improving the build Hi, I think the current download is pretty big. Is there is good reason that we do not use something like maven. Or at least, if you do not like maven, ivy to share dependencies between projects. Also enforces you to use libraries that are generally available. I would also love to have the (unit)tests closer to the actual code, hard to locate the tests at the moment Would like to hear the thoughts of the developers regards -- Jettro Coenradie http://www.gridshore.nl -- Jettro Coenradie http://www.gridshore.nl -- Jettro Coenradie http://www.gridshore.nl -- Jettro Coenradie http://www.gridshore.nl
[jira] Updated: (CONNECTORS-91) Making the initialization commands more useable
[ https://issues.apache.org/jira/browse/CONNECTORS-91?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jettro Coenradie updated CONNECTORS-91: --- Attachment: commandsPatch.patch the patch with an improvement for the commands Making the initialization commands more useable --- Key: CONNECTORS-91 URL: https://issues.apache.org/jira/browse/CONNECTORS-91 Project: Lucene Connector Framework Issue Type: Improvement Components: Framework core Reporter: Jettro Coenradie Fix For: LCF Release 0.5 Attachments: commandsPatch.patch At the moment LCF comes with some classes that can be used to run command line to interact with the system. Examples are DBCreate, DBDrop and LockClean. I wanted to create a class that rebuilds my complete environment. So dropping a database, creating a database, cleaning the synch folder, registering agents, etc. Due to the structure of the classes with all the logic in the main method, I could not easily reuse these classes. In the patch I submit with issue I have refactored the current solution in a better reuseable solution that can still be called command line. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Connector architecture question and suggestion
Hi, I am having a look at the connectors. At the moment to my opinion the classes for (all) connectors are to big. This is partly due to the way the interfaces are structured and partly due to the implementation of html in java. For example the RssConnector now has almost 6000 lines of code and the JdbcConnector has almost 2000 lines of code. To my opinion this can be improved by making separate components for presenting and configuring the connectors in the crawler-ui web project and for the part needed by the runner. Abstracting the html from the actual classes can help a lot as well. Maybe some utility methods to make creating these html pages easier is nice as well. I am willing to investigate this path further, but I'd like to have ideas of other developers. It would be nice to know if others feel like this can be improved as well. It might be interesting to look at a technique like wicket for the ui part. Than you can package the html code together with the java code in one jar. No difficult repackaging is required and you can still create nice interfaces. I also read that others want to have a look at something like velocity, of course that can be a valid option as well. So what is your opinions about it? regards Jettro Coenradie http://www.gridshore.nl
[jira] Commented: (CONNECTORS-91) Making the initialization commands more useable
[ https://issues.apache.org/jira/browse/CONNECTORS-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898857#action_12898857 ] Jettro Coenradie commented on CONNECTORS-91: If you feel this is the way to go, I will change the other classes that have a main method as well. Making the initialization commands more useable --- Key: CONNECTORS-91 URL: https://issues.apache.org/jira/browse/CONNECTORS-91 Project: Lucene Connector Framework Issue Type: Improvement Components: Framework core Reporter: Jettro Coenradie Fix For: LCF Release 0.5 Attachments: commandsPatch.patch At the moment LCF comes with some classes that can be used to run command line to interact with the system. Examples are DBCreate, DBDrop and LockClean. I wanted to create a class that rebuilds my complete environment. So dropping a database, creating a database, cleaning the synch folder, registering agents, etc. Due to the structure of the classes with all the logic in the main method, I could not easily reuse these classes. In the patch I submit with issue I have refactored the current solution in a better reuseable solution that can still be called command line. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-19) Look into converting SOLR connector to use SolrJ java library
[ https://issues.apache.org/jira/browse/CONNECTORS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898859#action_12898859 ] Jettro Coenradie commented on CONNECTORS-19: We have a working solr connector that makes use of solr. This might be a good start. I might need to spend some time to make it run in the lcf build. We have a maven build to package it at the moment. If you are interested, let me know. Than I will spend the time on a patch. Look into converting SOLR connector to use SolrJ java library - Key: CONNECTORS-19 URL: https://issues.apache.org/jira/browse/CONNECTORS-19 Project: Lucene Connector Framework Issue Type: Improvement Components: Lucene/SOLR connector Reporter: Karl Wright Priority: Minor The SOLR connector currently uses its own multipart post code. It might be a good idea to convert it to use the SolrJ client api jar instead. This would require license confirmation, plus research to make sure there are no jar conflicts as a result, with any other connector. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CONNECTORS-19) Look into converting SOLR connector to use SolrJ java library
[ https://issues.apache.org/jira/browse/CONNECTORS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898859#action_12898859 ] Jettro Coenradie edited comment on CONNECTORS-19 at 8/16/10 7:45 AM: - We have a working solr connector that makes use of solrj. This might be a good start. I might need to spend some time to make it run in the lcf build. We have a maven build to package it at the moment. If you are interested, let me know. Than I will spend the time on a patch. was (Author: jettroc): We have a working solr connector that makes use of solr. This might be a good start. I might need to spend some time to make it run in the lcf build. We have a maven build to package it at the moment. If you are interested, let me know. Than I will spend the time on a patch. Look into converting SOLR connector to use SolrJ java library - Key: CONNECTORS-19 URL: https://issues.apache.org/jira/browse/CONNECTORS-19 Project: Lucene Connector Framework Issue Type: Improvement Components: Lucene/SOLR connector Reporter: Karl Wright Priority: Minor The SOLR connector currently uses its own multipart post code. It might be a good idea to convert it to use the SolrJ client api jar instead. This would require license confirmation, plus research to make sure there are no jar conflicts as a result, with any other connector. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
improving the build
Hi, I think the current download is pretty big. Is there is good reason that we do not use something like maven. Or at least, if you do not like maven, ivy to share dependencies between projects. Also enforces you to use libraries that are generally available. I would also love to have the (unit)tests closer to the actual code, hard to locate the tests at the moment Would like to hear the thoughts of the developers regards -- Jettro Coenradie http://www.gridshore.nl
[jira] Commented: (CONNECTORS-91) Making the initialization commands more useable
[ https://issues.apache.org/jira/browse/CONNECTORS-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898925#action_12898925 ] Jettro Coenradie commented on CONNECTORS-91: There should be no subtleties, I mainly moved code from the main method into a new method and indeed a bit of class-inheritance. Making the initialization commands more useable --- Key: CONNECTORS-91 URL: https://issues.apache.org/jira/browse/CONNECTORS-91 Project: Lucene Connector Framework Issue Type: Improvement Components: Framework core Reporter: Jettro Coenradie Fix For: LCF Release 0.5 Attachments: commandsPatch.patch At the moment LCF comes with some classes that can be used to run command line to interact with the system. Examples are DBCreate, DBDrop and LockClean. I wanted to create a class that rebuilds my complete environment. So dropping a database, creating a database, cleaning the synch folder, registering agents, etc. Due to the structure of the classes with all the logic in the main method, I could not easily reuse these classes. In the patch I submit with issue I have refactored the current solution in a better reuseable solution that can still be called command line. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Suggestion for a Hippo repository connector
I will do that, need to check for the libraries that Hippo needs, will create a report and come back with that. On Mon, Aug 16, 2010 at 2:52 PM, karl.wri...@nokia.com wrote: Connector contributions are always welcome, even if others are not using it as of yet. Can you create a ticket which describes the connector and its environment (including the security model the repository uses, and what authority connector is appropriate to use with it), and attach your connector as a patch? Thanks! Karl -Original Message- From: jettro.coenra...@gmail.com [mailto:jettro.coenra...@gmail.com] On Behalf Of ext Jettro Coenradie Sent: Monday, August 16, 2010 8:07 AM To: connectors-dev@incubator.apache.org Subject: Suggestion for a Hippo repository connector Hi, We have a basic hippo repository connection that we could make available for the LCF project. Are there people that would like to have such a connector? regards -- Jettro Coenradie http://www.gridshore.nl -- Jettro Coenradie http://www.gridshore.nl
Re: improving the build
We could use something like profiles in maven. That way you can easily configure which projects to compile and which not. That would include tests. I will have a look at it and come up with a proposal. On Mon, Aug 16, 2010 at 2:49 PM, karl.wri...@nokia.com wrote: Re maven: There is a wiki page describing the Maven dependencies; somebody needs to tackle this. If you want to volunteer, we'd love to hear your proposal. However, please remember that you really must be sure to retain the connector conditional compilation structure as is currently in place, for license reasons. Re unit tests: The Junit test code was actually placed carefully based on the above considerations. In other words, you can't run a test that requires connectors x,y,z unless those connectors have actually been built. Similarly, you may think in terms of testing a specific connector by including tests for that connector, but those tests cannot use any OTHER connectors or you will break the build, which is why any tests that use multiple connectors must be at the module level. Karl -Original Message- From: jettro.coenra...@gmail.com [mailto:jettro.coenra...@gmail.com] On Behalf Of ext Jettro Coenradie Sent: Monday, August 16, 2010 8:21 AM To: connectors-dev@incubator.apache.org Subject: improving the build Hi, I think the current download is pretty big. Is there is good reason that we do not use something like maven. Or at least, if you do not like maven, ivy to share dependencies between projects. Also enforces you to use libraries that are generally available. I would also love to have the (unit)tests closer to the actual code, hard to locate the tests at the moment Would like to hear the thoughts of the developers regards -- Jettro Coenradie http://www.gridshore.nl -- Jettro Coenradie http://www.gridshore.nl
[jira] Commented: (CONNECTORS-19) Look into converting SOLR connector to use SolrJ java library
[ https://issues.apache.org/jira/browse/CONNECTORS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898928#action_12898928 ] Jettro Coenradie commented on CONNECTORS-19: I will have a good look at the dependencies and the functionality. If satisfied, I will supply a patch that other can check as well. Look into converting SOLR connector to use SolrJ java library - Key: CONNECTORS-19 URL: https://issues.apache.org/jira/browse/CONNECTORS-19 Project: Lucene Connector Framework Issue Type: Improvement Components: Lucene/SOLR connector Reporter: Karl Wright Priority: Minor The SOLR connector currently uses its own multipart post code. It might be a good idea to convert it to use the SolrJ client api jar instead. This would require license confirmation, plus research to make sure there are no jar conflicts as a result, with any other connector. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Connector architecture question and suggestion
Oke, I will come up with a proposal for breaking up the components in a way that still enables us to easily keep the current adapters in their own structure. As for the UI testen, it is always a beast. We do have some experience with tools like Webdriver and others. Will have a look at that as well. On Mon, Aug 16, 2010 at 2:41 PM, karl.wri...@nokia.com wrote: I think that providing tools/help for implementing the UI pieces of connectors is a perfectly reasonable thing to do. However, I strongly believe that the UI components should remain described as part of the connector interfaces. Breaking their implementations out within individual connectors is also reasonable, but unless there is a compelling reason to refactor all the individual connectors in that way, I would hold off that project until there are better unit tests for the connector UI components. If you are willing to contribute such tests, I think going that way would be worth consideration. I'd like to see a more detailed proposal before I comment further. Thanks, Karl -Original Message- From: jettro.coenra...@gmail.com [mailto:jettro.coenra...@gmail.com] On Behalf Of ext Jettro Coenradie Sent: Monday, August 16, 2010 7:37 AM To: connectors-dev@incubator.apache.org Subject: Connector architecture question and suggestion Hi, I am having a look at the connectors. At the moment to my opinion the classes for (all) connectors are to big. This is partly due to the way the interfaces are structured and partly due to the implementation of html in java. For example the RssConnector now has almost 6000 lines of code and the JdbcConnector has almost 2000 lines of code. To my opinion this can be improved by making separate components for presenting and configuring the connectors in the crawler-ui web project and for the part needed by the runner. Abstracting the html from the actual classes can help a lot as well. Maybe some utility methods to make creating these html pages easier is nice as well. I am willing to investigate this path further, but I'd like to have ideas of other developers. It would be nice to know if others feel like this can be improved as well. It might be interesting to look at a technique like wicket for the ui part. Than you can package the html code together with the java code in one jar. No difficult repackaging is required and you can still create nice interfaces. I also read that others want to have a look at something like velocity, of course that can be a valid option as well. So what is your opinions about it? regards Jettro Coenradie http://www.gridshore.nl -- Jettro Coenradie http://www.gridshore.nl
