On May 30, 2015, at 12:34 PM, Shameera Rathnayaka <shameerai...@gmail.com> wrote: > > Hi Suresh, > > I do not think bundling all core into one module is a good idea. When there > is distinct enough functionality why combine? There are advantages with > micro-service architecture both in terms of rapid evolution as well as ease > of light weight deployments. For instance lets take API server and > Orchestrator example. API server gets hit for all the calls, but only a > fraction of them are intended for Orchestrator. Even in your approach of > starting different services from one module, the dependencies are all class > loaded (unless we put sophistication in startup scripts). > > If two modules have significant overlapping functionality then + 1 to discuss > those as case by case and combine them as necessary. But can you explain a > little more on why you want to combine distinct functional modules? > > A good example is Apache Spark, it is a rapid evolving project, and it also > has different major components ( Sprak SQL, Spark Streaming, MLlib and > GraphX) but all share one core component[1]. No doubt they need light > weight deployments more than us, as they have higher user community.
Spark is not the right comparison for this discussion. I have been a spark incubation mentor and have been following the code organization since its early days. All of the spark components you mentions rely on the core. Let me step back and ask, what is the problem you are trying to solve? We sure need to cleanup modules and it is time to re-look at the component organization. But what do you want to really achieve by combining all the core into monolithic components. It took an effort to cleanly separate functionality so they can evolve and can be improved independently. > So why do we make our repository bulky with modules unless it doesn't provide > any considerable advantage. > If we really want to separate distribution bundle for each component ( > apiServer , Orchestrator and Gfac ) let's use different bin.xml files to > do it instead of using different modules. But reality is we only use all in > one distribution. Once the monitoring is fixed to use messaging, we really need to decouple the component deployments. Yes there is a considerable advantage. Each components has different quality of service requirements. A production platform has to load balance and scale horizontally. And thats different for different component. The all in one bundle has 300+ jars, but API server and orchestrator when independent will have around 50 or so jars. When I want to deploy api server and orchestrator there is significant different in small light weight components vs one monolithic core. Another problem is component evolution. Lets say there is a production deployment running 1.1.4 version. Lets say the single job execution is stable enough and there is a 6 month focused effort on workflow. Say the master moves to 1.2.8 with all the changes to workflow and only few to single application execution. We can more comfortably upgrade is they are cleanly separated modules. But if it is one core and so many changes to it (even though technically they are to different classes, the perception will remain), the upgrades will get behind. Bottom line I am + 1 for cleaning up the modules. Past few years we have been moving towards micro service architectures and your suggestions will reverse this back to monolithic architecture. I am -1 for this change in direction. Suresh > > Thanks, > Shameera. > > > Suresh > >> On May 29, 2015, at 11:29 AM, Suresh Marru <sma...@apache.org >> <mailto:sma...@apache.org>> wrote: >> >> + 1. >> >> I was planning to bring up this issue also. Probably it will not address >> what you are raising, but here is a tree output from airavata labs code I >> was toying with locally. I did not yet compare it with what you proposed, I >> will do so later today. >> >> ├── airavata-api >> │ ├── airavata-api-interface-descriptions >> │ ├── airavata-api-java-stubs >> │ ├── airavata-api-server >> │ ├── airavata-data-models >> │ ├── api-security-manager >> ├── clients >> │ ├── airavata-client-cpp-sdk >> │ ├── airavata-client-java-sdk >> │ ├── airavata-client-php-sdk >> │ ├── airavata-client-python-sdk >> │ ├── airavata-sample-examples >> │ └── airavata-xbaya-gui >> ├── components >> │ ├── commons >> │ ├── component-interface-descriptions >> │ ├── component-services >> │ │ ├── credential-store-service >> │ │ ├── orchestrator-service >> │ │ ├── task-executor-service >> │ │ └── workflow-interpreter-service >> │ ├── component-clients >> │ │ ├── credential-store-client >> │ │ ├── orchestrator-client >> │ │ ├── task-executor-client >> │ │ ├── workflow-interpreter-client >> │ │ └── messaging >> │ ├── task-adaptors >> │ │ ├── compute >> │ │ └── data-movement >> │ ├── registry >> │ │ ├── app-catalog >> │ │ ├── experiment-catalog >> │ │ └── resource-catalog >> │ └── workflow-interpreter >> ├── distribution >> ├── integration-tests >> >> >> >>> On May 29, 2015, at 10:15 AM, Shameera Rathnayaka <shame...@apache.org >>> <mailto:shame...@apache.org>> wrote: >>> >>> Hi Devs, >>> >>> As we are using different modules to package different type of >>> functionalities, which will help us to maintain loosely couple codes. Now >>> the project has 49 leaf module ( one to hit half century :) ). If we allow >>> project to grow this way, having too fine grain modules will be huge >>> headache in future. IMO we should clean this ASAP before it become really >>> mess. Actually we half way there, I experienced cyclic dependency issues >>> when I was writing workflow implementation and email monitoring. Please see >>> the modules in current repo below. >>> >>> <module-name> ( <num of child modules> ) >>> >>> modules ( 43 ) >>> app-catalog ( 2 ) >>> commons ( 1 ) >>> configurations ( 2 ) >>> credential-store ( 3 ) >>> distribution ( 8 ) >>> gfac ( 10 ) >>> integration test ( 1 ) >>> messaging ( 2 ) >>> orchestrator ( 3 ) >>> registry ( 3 ) >>> security ( 1 ) >>> server ( 1 ) >>> test-suit ( 1 ) >>> workflow ( 1 ) >>> workflow-modal ( 3 ) >>> xbaya ( 1 ) >>> airavata-api ( 5 ) >>> tools ( 1 ) >>> >>> Most of the current modules have interfaces and implementations together, >>> but this violate our main goal which reduce inter module dependencies. >>> Following is what I am suggesting, WDYS? >>> >>> core { has all core interfaces and basic classes of gfac-core , >>> orchestrator-core , message-core , monitor core, registry core, >>> workflow-core} >>> service - all thrift services and service handlers >>> orchestrator - orchestrator specific classes >>> gfac >>> SSH >>> BES >>> Local >>> message - amqp implemention >>> distribution >>> XBaya >>> server - { use different mode input to start server as orchestrator , >>> Gfac or/and api-server } >>> commons >>> registry >>> app-catalog >>> security >>> Workflow >>> XBaya-gui >>> Integration-test >>> >>> Thanks, >>> Shameera. >> > > > > > -- > Best Regards, > Shameera Rathnayaka. > > email: shameera AT apache.org <http://apache.org/> , shameerainfo AT > gmail.com <http://gmail.com/> > Blog : http://shameerarathnayaka.blogspot.com/ > <http://shameerarathnayaka.blogspot.com/>
