I chatted with Venkat earlier today on #tuscany about changes he proposed to contribution processing. This is input to the wider, what do we do about contribution processing and workspace debate....
<Venkat <irc://freenode/Venkat,isnick>>I am trying to clean up the contribution service imple a bit <slaws>so you want to talk a bit about your plans? <Venkat <irc://freenode/Venkat,isnick>>yes <slaws>ok- we could also take a look together at the changes sebastien has made to contribution processing <Venkat <irc://freenode/Venkat,isnick>>basically I am wondering if contribution ever needs to depend on 'assembly' or 'policy' or 'definitions' module <Venkat <irc://freenode/Venkat,isnick>>for now am starting with cleaning up for decoupling the 'definitions' dependency <slaws>i'm with you there. It should depend on as little as possible. Not sure we can mitigate assembly but lets see <Venkat <irc://freenode/Venkat,isnick>>right now... we are doing a bit of preprocessing of composite files to add the 'applicablePolicySets' to various SCA Artifcats <slaws>y - i saw that <Venkat <irc://freenode/Venkat,isnick>>for this we need to take note of all the PolicySets that are getting contributed... <Venkat <irc://freenode/Venkat,isnick>>i.e. when a defintions.xml is read... we would like to dive in and weed out the policysets <Venkat <irc://freenode/Venkat,isnick>>then these policysets will have to be processed against every composite that is read... and which ever policyset applies <Venkat <irc://freenode/Venkat,isnick>>to which ever sca artifact in the composite... that policyset's name has to get into the 'applicablePolicySets' attribute <Venkat <irc://freenode/Venkat,isnick>>so all of this is now being done in the Contribution Service <Venkat <irc://freenode/Venkat,isnick>>so the contribution service needs to know when its a definition that was read... <Venkat <irc://freenode/Venkat,isnick>>and then needs to weed out the policysets out of it.. and then when processing composites it has to use this.. <Venkat <irc://freenode/Venkat,isnick>>so that brings in the dependencies.. i.e. contribution serivce needs to know about SCADefinitions, PolicySet and so on <Venkat <irc://freenode/Venkat,isnick>>am wondering if I could decouple this and tear it out of the ContributionService.. <Venkat <irc://freenode/Venkat,isnick>>so here is what I am thinking of.. <slaws>ok- go ahead <Venkat <irc://freenode/Venkat,isnick>>when artifacts are read by the contribution service... it coudl publish events to interested listners... <Venkat <irc://freenode/Venkat,isnick>>it just about sends events that has the URL that was processed and the model that was read... (need not know anything of the actually model type) <Venkat <irc://freenode/Venkat,isnick>>the runtime listens to these events and if there are defintions read.. it weeds out policysets... and aggregates it <Venkat <irc://freenode/Venkat,isnick>>the composite preprocessing will move to the CompositeDocumentProcessor <Venkat <irc://freenode/Venkat,isnick>>where in before the reading of the document, the processor will preprocess and add the applicable policysets.. <Venkat <irc://freenode/Venkat,isnick>>now how does the CompositeDocumentProcessor get hold of the policysets... since its the runtime that creates this processor <Venkat <irc://freenode/Venkat,isnick>>it should be fine for it to inject the policySets list that it is aggregating... <Venkat <irc://freenode/Venkat,isnick>>the only ugly thing here is probably the PolicySet list.. which is a sort of sink... a reference to it <Venkat <irc://freenode/Venkat,isnick>>is passed to the CompositeDocumentProcessor during its creation.. and then during the contribution processing.. its the same sink <Venkat <irc://freenode/Venkat,isnick>>into which the runtime dumps the policysets weeded out.. <Venkat <irc://freenode/Venkat,isnick>>so thats for what is running in my mind... for this specific decoupling... over to your thoughts :) |<-- <slaws>ok- sounds like you are thinking about a much more modular approach to definitions and policy processing. Can we enumerate the individual parts of that processing here so I can see them clearly.. <slaws>let me make a list based on what you have just said... <slaws>1 - read definitions.xml <slaws>2 - aggregate policy sets <slaws>3 - apply policy sets to composite model <Venkat <irc://freenode/Venkat,isnick>>(to composite xml) <slaws>ok - 3 - apply policy sets to composite.xml <slaws>i didn't quite understand the bit about.... <slaws> <Venkat> into which the runtime dumps the policysets weeded out.. <slaws>can you say a little more <Venkat <irc://freenode/Venkat,isnick>>its the "2 - aggregate policysets" ... its right now done by the contribution service... I want to move that to the runtime <Venkat <irc://freenode/Venkat,isnick>>so instead of the contrib svc doing it. <Venkat <irc://freenode/Venkat,isnick>>it will simply pass a SCADefintions model to the runtime... as part of an event <Venkat <irc://freenode/Venkat,isnick>>the runtime will look this SCADeinitios and aggregate the PolicySets alone <slaws>and what are the sources of the policy sets that are being aggregated?... <Venkat <irc://freenode/Venkat,isnick>>multiple definitions.xml files... that are coming in thro contributions <slaws>should 2 read 2 - aggregate policy sets from all of the defintions files found in all contributions <Venkat <irc://freenode/Venkat,isnick>>yes <slaws>ok - so our list is now.... <slaws>1 - read definitions.xml from all contributions <slaws>2 - aggregate policy sets from all of the defintions files found in all contributions and read in 1 <slaws>3 - apply policy sets to each composite.xml as it is processed <slaws>is that right? <Venkat <irc://freenode/Venkat,isnick>>right... ahead of 1 will be a step where we do '2' for definitions.xml that are coming from our various modules... <Venkat <irc://freenode/Venkat,isnick>>but that will be done by the runtime itsefl.. <slaws>ok so the full list should be <Venkat <irc://freenode/Venkat,isnick>>so there is no looking up the contribution service for this part.. <slaws>1 - read all definitions.xml files found on the system classpath <slaws>2 - read definitions.xml from all contributions <slaws>3 - aggregate policy sets from all of the defintions files read in steps 1 and 2 <slaws>4 - apply policy sets to each composite.xml as it is processed <slaws>sound ok? <Venkat <irc://freenode/Venkat,isnick>>right... (just the one comment that for '1' I will put in the 'provider' mechanism that Sebastien had suggested.. but to do that I want to first clean this up to get some circular dependencies out of the way) <slaws>y - I was being purposely vague for 1 as I know we have discussed that on the list and changes are afoot <Venkat <irc://freenode/Venkat,isnick>>(but this is a detail we don't have to worry in this context) <slaws>y - agree <Venkat <irc://freenode/Venkat,isnick>>ok :) <slaws>so can we take 2, 3 and 4 in turn and look at you porposals <Venkat <irc://freenode/Venkat,isnick>>yes <slaws>ok - so 2 is all about how to remove the necessity of the contribution service to do specific definitions.xml processing <slaws>well 2 and 3 <slaws>can we take a slight diversion here and look at what sebastien did in the workplace processing <slaws>take a look at this file.. <slaws> http://svn.apache.org/repos/asf/incubator/tuscany/java/sca/modules/workspace-impl/src/main/java/org/apache/tuscany/sca/workspace/processor/impl/ContributionContentProcessor.java <Venkat <irc://freenode/Venkat,isnick>>ok am there <slaws>looking at the file he has created these scanners that he uses to produce the artifact list... <slaws>each artifact is then read in turn using the artifactProcessor <slaws>and its subsequently resolved using the model resolver <slaws>so this is a pretty generic loop <slaws>but take a look at this file.... <Venkat <irc://freenode/Venkat,isnick>>right <slaws> http://svn.apache.org/repos/asf/incubator/tuscany/java/sca/modules/workspace-impl/src/main/java/org/apache/tuscany/sca/workspace/processor/impl/ContributionInfoProcessor.java <slaws>in this case he is looking for specific files in the contribution <slaws>they are in a specific place in the contribution in this case but I'm wondering whether you can use a similar idea to isolate the definitions.xmlfiles in the artifact list and process them rather than having to invent some listener/callback mechanism to do the same <slaws>i guess what I'm saying is that it seems simpler (although more processor intensive) to look across the artifacts list a number of times and pick out the bits you want to process rather than having a single processor that does it all with a listener mechanism that calls back when it finds artifacts that have had listeners registered <slaws>am I making sense? <Venkat <irc://freenode/Venkat,isnick>>but this will only solve the 'reading' problem.. I will know that a definitions.xml has been read... what happens after that.. <Venkat <irc://freenode/Venkat,isnick>>which is the aggregation.. <Venkat <irc://freenode/Venkat,isnick>>of policysets <Venkat <irc://freenode/Venkat,isnick>>and the reading is pretty much fine in the Contribution Serivce... <slaws>ok so I was imagining that you have a readDefinitionsAndAggregatePolicySets(CntributionURL, PolicySetsCollection) that reads the definitions files and aggreagates the policysets. This is independent of other contribution processing <Venkat <irc://freenode/Venkat,isnick>>where would you imagine this method to be -->| <irc://freenode/%23tuscany> <slaws>modules/contributio-defintions, modules/defnitions or even modules/host-embedded <Venkat <irc://freenode/Venkat,isnick>>so if this method was in the contribution service... and yes.. the contirbution service could just about dump the defintions model object into a list without really having to know about SCADefinitions but just relying on the fact that the artifact file is defintions.xml <slaws>i'm going to step out for a few mins while I ponder the "where" question. back shortly <Venkat <irc://freenode/Venkat,isnick>>but then where will the PolicySets be weeded out of the SCA defintions ... before the composites start getting processed .. <Venkat <irc://freenode/Venkat,isnick>>sure.. please carry on <Venkat <irc://freenode/Venkat,isnick>>ping me when you are back <slaws>ok- am back you there? <Venkat <irc://freenode/Venkat,isnick>>yes Simon :) <slaws>ok - in ReallySmallRuntime today it constructs a compositeProcessor and then just provides it on request (it also does step 1 from out list but I'm ignoring that for the moment)... <slaws>ooops CompositeProcessor -> ContributionService <slaws>then whatever runtime SCADomain, Node etc, uses RSR, grabs the ContributionService and calls contribute <Venkat <irc://freenode/Venkat,isnick>>right <slaws>i'm imagining a situation where SCADomain, Node, whatever does more work and doesn't use a black box ContributionService <Venkat <irc://freenode/Venkat,isnick>>to process contributions ? <slaws>Y - so It would go down our list of items in turn running appropriate methods to make each of them happen. <Venkat <irc://freenode/Venkat,isnick>>yes... <slaws>The list of aggregated policies would result from step 3 . It could hold onto the list provide it to whatever bit of the runtime requires them <slaws>the trick bit seems to be step 4 <slaws>where you need to get at the composite as a dom so you can apply xpath <slaws>to match policies <Venkat <irc://freenode/Venkat,isnick>>yes.. thats the key actually... its like this.. <Venkat <irc://freenode/Venkat,isnick>>1. read up all contributions other than composite files... <Venkat <irc://freenode/Venkat,isnick>>2. when definitions.xml are read aggregate policysets. <Venkat <irc://freenode/Venkat,isnick>>3. read composites and preprocess them applying the read policysets... <Venkat <irc://freenode/Venkat,isnick>>so there is a bit of orchestration here... <Venkat <irc://freenode/Venkat,isnick>>you push the composite processing to the end... <Venkat <irc://freenode/Venkat,isnick>>to make sure you have all the policysets for the domain read <slaws>y, i see, and there are (at least) two ways of viewing these steps... <slaws>A - Top down - drive everything based on reading the contribution once and process things appropriately as you come across them <slaws>B - bottom up - extract things you are interested in from the contribution and process them appropriately. <slaws>am just making these terms up ;-) <slaws>so I think B works well for sca-contribution.xml, definitions.xml, *.composite, i.e. the things we can reasonably expect to be there <slaws>A works well for everything else <slaws>where we don't know what to expect and just have to deal with it when we find it <slaws>I'm saying this because of what you just said about needing to be carefult about how the processing is ordered <slaws>so I can imagine some very specific blocks of processing. <slaws>i) locate all the contributions that are expected to run <slaws>ii) for each contribution read all the contribution meta-data <slaws>iii) for each contribution locate all the artifacts <slaws>oops iii) for each contribution read all the definitions.xml files and aggregate policy sets with those already read <slaws>iv) for each contribution read all the artifacts <slaws>v) find the .composite files amongst the artifacts, preprocess to apply policy sets and then read into a model <slaws>change to iv) <slaws>iv) for each contribution read all the artifacts into models excluding .composite files <slaws>and I imagine these steps happening independently of the current contributionService <Venkat <irc://freenode/Venkat,isnick>>so what would the contributionService ever do .. ? <slaws>i think we could deprecate it <Venkat <irc://freenode/Venkat,isnick>>alright.. so which ever module that does this... <Venkat <irc://freenode/Venkat,isnick>>its a question of whether it would tolerate dependencies on defintions and policy modules <Venkat <irc://freenode/Venkat,isnick>>if you are saying.. <Venkat <irc://freenode/Venkat,isnick>>that there will be a runtime that packs all of this within... <Venkat <irc://freenode/Venkat,isnick>>then I suppose its ok... <Venkat <irc://freenode/Venkat,isnick>>becuase its ok for a runtime to know about SCADefinitions or PolicySets... <slaws>the general model I have in my mind is that tuscany provides a number of independent (as possible) modules that provide independent services are, to use your term, orchestrated by a (runtime) modules that brings it all together.. <slaws>this is slightly different what we have at the moment where, in some cases, we have multiple levels of this orchestration type activity <slaws>for example, in this contribution case a runtime does a few things and then calls the contributionService which does a few more things. The runtime can break into the contribution service to modify the way that the independent parts of the contribution process are orchestrated <slaws>the runtime can break -> the runtime can't break -->| <Venkat <irc://freenode/Venkat,isnick>>hi.. am back <Venkat <irc://freenode/Venkat,isnick>>(sorry had to step out a bit) <slaws>ok <Venkat <irc://freenode/Venkat,isnick>>yes.. <Venkat <irc://freenode/Venkat,isnick>>read up what you said <Venkat <irc://freenode/Venkat,isnick>>so what you are saying is.. if events were published it would mean something from outside can hook into this internal <Venkat <irc://freenode/Venkat,isnick>>of the contirbution service ? <slaws>well I wasn't advocating publishing events. I was saying that the individual steps should be accessible so that you can hook into them without needing events <slaws>but, yes, the basic idea is to open up the ContributionService <Venkat <irc://freenode/Venkat,isnick>>you mean have more granular methods on that service interface ? <slaws>y <slaws>so if you ask me how we get from where we are now to what I was talking about (assuming of course we want to go there :-) then we could take a first step of exposing more granular methods. Then separate the methods into the modules where they belong <slaws>the first step won't address you dependency issue of course <slaws>but maybe it is best to really understand what the separate steps are by separating them out as methods before moving things around <Venkat <irc://freenode/Venkat,isnick>>right... I guess it could be on lines with the list you mentioned above.. which is above different ways of reading artifcats in a contribution <Venkat <irc://freenode/Venkat,isnick>>so every runtime can orchestrate how it wants to do it... <Venkat <irc://freenode/Venkat,isnick>>so I see this is not going to make it to 1.2... and then its best to attempt anything of this sort with the workspace efforts since that is going to at some point come and stay ? <Venkat <irc://freenode/Venkat,isnick>>in this discussion I managed to 'see' one more option of achieving what I want... <Venkat <irc://freenode/Venkat,isnick>>which will be to simply pass the 'sink' as a generic list.. so that contribution service need not know SCADefn type.. <Venkat <irc://freenode/Venkat,isnick>>so I will give this a shot .. <Venkat <irc://freenode/Venkat,isnick>>as its not going to change things much <slaws>ok - agreed that it won't make 1.2
