If thats the case then we are good to go with the current model. I will remove the configuration files from src/test/resources, add the server/client configuration module as a maven dependency in test scope and run the tests. If everything works fine I will commit this change.
myproxy credentials are read through our ServerSettings class. If these details need to be overridden from values set at system properties, I can update the ServerSettings class to check the system properties first before checking the server/client property file. wdyt? On Fri, Apr 4, 2014 at 11:14 AM, Lahiru Gunathilake <glah...@gmail.com>wrote: > Hi saminda, > > For unit tests we are only going to change credential related parameters > to check certain funcationality which we cannot test with localhost (ex: > pbs related stuff, slurm related stuff etc). In that case we only need to > change credentials which can be read through system properties (these > properties can be given in the maven command). In this case we can unify > all the test cases to read same properties and all the grid tests should > run with one configuration change (in command). > > If we are only changing credentials during unit tests, we should be good > with current model I guess. > > > On Fri, Apr 4, 2014 at 1:02 PM, Saminda Wijeratne <samin...@gmail.com>wrote: > >> >> >> A new properties file (say "custom.properties") would be introduced >> containing only the settings that are different from the shipped version of >> the server or client properties file. At runtime the Settings class will >> load the server/client properties file as usual and afterwards merge with >> the settings defined in custom.properties file. Each module will have its >> own custom.properties file defined for unit tests if they have settings >> that would different compared to the default valued. Thus the >> custom.properties file will act overloading and overriding the default >> server/client properties file in the classpath. wdyt? >> > if we have multiple modules, before running tests we have to modify every > place (because we cannot commit credentials to the repo, and at this point > we only need to change credentials). > > WDYT ? > >> >> >> >> On Fri, Apr 4, 2014 at 2:18 AM, Saminda Wijeratne <samin...@gmail.com>wrote: >> >>> hi devs, >>> >>> I refactored the trunk code to link airavata-server.properties and >>> airavata-client.properties from 2 maven modules now included in the trunk. >>> modules/configurations/server >>> modules/configurations/client >>> >>> airavata-server.properties is used by the "airavata-standalone-server" >>> module (@ modules/server) and in the server distribution. >>> airavata-client.properties was used in, >>> modules/orchestrator/orchestrator-client-sdks >>> modules/xbaya-gui >>> modules/distribution/xbaya-gui >>> modules/distribution/airavata-client >>> modules/airavata-client >>> airavata-api/airavata-client-sdks/java-client-samples >>> >>> However I was not able to carry out similar merge for the properties >>> files used in the tests since I kept hitting on broken unit tests. I will >>> try to fix them first along with the issue reported by lahiru [2]. >>> >>> Saminda >>> >>> 1. https://issues.apache.org/jira/browse/AIRAVATA-1121 >>> 2. https://issues.apache.org/jira/browse/AIRAVATA-1119 >>> >>> >>> >>> On Mon, Mar 31, 2014 at 6:40 AM, Lahiru Gunathilake >>> <glah...@gmail.com>wrote: >>> >>>> I think we already have it in gfac-schemas and using tool like this[1], >>>> we can quickly generate forms without worrying about changing them when the >>>> schema changes. >>>> >>>> But I think we are going to do a new Application Catalog and once its >>>> done we can use some tool to do the same thing. >>>> >>>> [1]https://code.google.com/p/xsd-web-forms/ >>>> Regards >>>> Lahiru >>>> >>>> >>>> On Mon, Mar 31, 2014 at 9:31 AM, Sachith Withana >>>> <swsach...@gmail.com>wrote: >>>> >>>>> Why don't we have some kind of Util module to store all these config >>>>> files and have a rest-kind of class which acts as the middle layer between >>>>> airavata modules and config files? >>>>> All the config files are in one place, have a programmatic method of >>>>> accessing them. >>>>> >>>>> IMO we should have a different set of config files for the Integration >>>>> tests, >>>>> 1. It would test the system using the thrift 'client' using external >>>>> configurations files >>>>> 2. since it is a main entry point for a gateway/airavata dev, it will >>>>> show a clearer image on which configurations we should use when we are >>>>> using the client (it would only be the URLs and the ports for now). >>>>> 3. It would eliminate any airavata-internal dependencies for the >>>>> integration tests. >>>>> >>>>> >>>>> On Mon, Mar 31, 2014 at 1:16 AM, Saminda Wijeratne <samin...@gmail.com >>>>> > wrote: >>>>> >>>>>> It depends upon the test. If someone adds or updates configuration >>>>>> data related to the correct functioning of the system or a component it >>>>>> is >>>>>> very likely that those configuration needs to be present for unit tests >>>>>> as >>>>>> well in case the unit tests depends upon those other components. Thus it >>>>>> becomes necessary to update all the test configuration files (to be on >>>>>> the >>>>>> safe side). >>>>>> On Mar 30, 2014 8:29 PM, "Lahiru Gunathilake" <glah...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Saminda, >>>>>>> >>>>>>> Ideally even at this point we should be able to remove all the >>>>>>> configuration files from src/main/resources but for tests we might need >>>>>>> the >>>>>>> files in src/test/resources. >>>>>>> >>>>>>> good thing about this is we have all the configuration in one place >>>>>>> (airavata-server.properties) but bad thing is we have to keep it in each >>>>>>> module's test configuration. >>>>>>> >>>>>>> Is this really a problem ? >>>>>>> >>>>>>> Regards >>>>>>> Lahiru >>>>>>> >>>>>>> >>>>>>> On Sun, Mar 30, 2014 at 9:59 PM, Saminda Wijeratne < >>>>>>> samin...@gmail.com> wrote: >>>>>>> >>>>>>>> In another related note I found few places which has additional >>>>>>>> property file types present in our trunk, some examples are, >>>>>>>> >>>>>>>> tools/job-monitor/src/main/resources/gsissh.properties >>>>>>>> tools/job-monitor/src/test/resources/monitor.properties >>>>>>>> tools/phoebus-integration/src/main/resources/service.properties >>>>>>>> modules/commons/gfac-schema/src/main/resources/datatype.properties >>>>>>>> >>>>>>>> modules/commons/workflow-tracking/src/main/resources/log4j.properties >>>>>>>> >>>>>>>> modules/orchestrator/airavata-orchestrator-service/src/test/resources/orchestrator.properties >>>>>>>> >>>>>>>> modules/ws-messenger/messagebroker/src/test/resources/unit_tests.properties >>>>>>>> modules/test-suite/src/test/resources/unit_test.properties >>>>>>>> modules/xbaya-gui/src/main/resources/airavata-client.properties >>>>>>>> modules/xbaya-gui/src/test/resources/airavata-server.properties >>>>>>>> modules/gfac/gfac-core/src/main/resources/errors.properties >>>>>>>> modules/gfac/gfac-core/src/main/resources/service.properties >>>>>>>> modules/gfac/gfac-core/src/test/resources/logging.properties >>>>>>>> modules/gfac/gfac-core/src/test/resources/airavata-client.properties >>>>>>>> >>>>>>>> I remember we raised an issue relating to having component property >>>>>>>> files but not coming to a conclusion regarding it. Any thoughts about >>>>>>>> it? >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Mar 27, 2014 at 9:23 PM, Saminda Wijeratne < >>>>>>>> samin...@gmail.com> wrote: >>>>>>>> >>>>>>>>> One way which so far seems to work is to define 2 maven modules >>>>>>>>> airavata-server-configuration and airavata-client-configurations as >>>>>>>>> jar >>>>>>>>> artifacts where each will have the respective configuration files. >>>>>>>>> Then use >>>>>>>>> them as dependencies for the modules which require the relevant >>>>>>>>> configurations. For the binary distributions we will have to use the >>>>>>>>> assembly-plugin and/or the dependency-plugin to copy the configuration >>>>>>>>> files in to the correct location in the distribution directory. >>>>>>>>> >>>>>>>>> I'm still working on how this solution can be adhered when running >>>>>>>>> unit tests. At the moment the the duplicated configuration files in >>>>>>>>> src/test/resources only has a few changes compared to the distributed >>>>>>>>> configurations. Any thoughts? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Mar 26, 2014 at 10:22 PM, Saminda Wijeratne < >>>>>>>>> samin...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> A little update on the efforts on this... >>>>>>>>>> >>>>>>>>>> - Like Amila suggested, ServerSettings now by default >>>>>>>>>> supports parameterization >>>>>>>>>> - settings can be added/updated by passing the new settings >>>>>>>>>> via command line (and incidentally passing them to the static >>>>>>>>>> func. >>>>>>>>>> ServerMain.main(...)). >>>>>>>>>> - $ ./airavata-server.sh --myproxy.user=saminda >>>>>>>>>> --myproxy.pass=pass123 >>>>>>>>>> - Introduced ServerSettings.mergeSettings...(...) >>>>>>>>>> functions to help add/update settings through Maps, command line >>>>>>>>>> args, >>>>>>>>>> additional prop files. >>>>>>>>>> >>>>>>>>>> The issue that I'm still facing is the duplication of the >>>>>>>>>> airavata-server.properties configuration file in the code base. Last >>>>>>>>>> time I >>>>>>>>>> checked copies of it is spread out at 14 locations in the trunk (for >>>>>>>>>> running junit tests, in standalone server dist, in integration >>>>>>>>>> tests, in >>>>>>>>>> sample gateway). >>>>>>>>>> >>>>>>>>>> - config files in a single location in the trunk and refer to >>>>>>>>>> them using relative paths: >>>>>>>>>> - done using the maven-builder-helper plugin. >>>>>>>>>> - But it seems the IDEs doesn't recognize resources >>>>>>>>>> outside the project folder. >>>>>>>>>> - "git subtrees" (which allows two way code sharing) similar >>>>>>>>>> to "svn externals": >>>>>>>>>> - the limitations in it (configs needs to be in a different >>>>>>>>>> repo, subtree will be merged on the root, cannot specifically >>>>>>>>>> select which >>>>>>>>>> code to share?), it seems its not a viable option. >>>>>>>>>> >>>>>>>>>> any ideas? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Mar 6, 2014 at 1:55 PM, Amila Jayasekara < >>>>>>>>>> thejaka.am...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Saminda, >>>>>>>>>>> >>>>>>>>>>> I guess the best thing is to create template configuration files >>>>>>>>>>> in a central location and replace needed variables within the test >>>>>>>>>>> initialization and place generated config files within the target >>>>>>>>>>> directory >>>>>>>>>>> of the test case. >>>>>>>>>>> >>>>>>>>>>> E.g :- registry.url = ${registry.url} >>>>>>>>>>> >>>>>>>>>>> ${registry.url} is replaced within the test case with >>>>>>>>>>> appropriate value. Then we dont need to duplicate configuration >>>>>>>>>>> files in >>>>>>>>>>> everywhere. Also when a configuration is updated we only need to >>>>>>>>>>> change in >>>>>>>>>>> a single place. >>>>>>>>>>> You may encapsulate template parameter replacing logic into a >>>>>>>>>>> common util class so that each test case can just invoke the logic. >>>>>>>>>>> >>>>>>>>>>> Further we should have a single place to read all configurations >>>>>>>>>>> and all subcomponents must go through this configuration component >>>>>>>>>>> to get >>>>>>>>>>> config values. Otherwise it will be hard to maintain the >>>>>>>>>>> configuration >>>>>>>>>>> reading code. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> Amila >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Mar 6, 2014 at 1:38 PM, Saminda Wijeratne < >>>>>>>>>>> samin...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> There are several places which we need the >>>>>>>>>>>> airavata-server.properties and airavata-client.properties files to >>>>>>>>>>>> be >>>>>>>>>>>> present in order to run tests and standalone servers resulting in >>>>>>>>>>>> replication. Any idea how we should handle this? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> System Analyst Programmer >>>>>>> PTI Lab >>>>>>> Indiana University >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> Sachith Withana >>>>> >>>>> >>>> >>>> >>>> -- >>>> System Analyst Programmer >>>> PTI Lab >>>> Indiana University >>>> >>> >>> >> > > > -- > System Analyst Programmer > PTI Lab > Indiana University >
