Hi Peter,
Thank you for your reply first, please see the following comments
2011/7/28 Peter Kriens <[email protected]>:
> It is good that you use a real project now. What I like to see how many
> "bundles" we have after the first round, where you group strongly connected
> packages. This should already simplify because the nr of entities will be
> smaller.
>
> I need to see a real case. Step 1 is ok, but maybe you can add merging of
> bundles that have identical usesExternal (imports).
Yes, I do merge the bundles that have identical usesExternal
(imports), i described in the former mail,
We define sameUE: it menas the number of two bundles have same
usedExternal external package rely elements.
boolean condition2 = 3 * sameUE >= one.usedExternalList.size() +
two.usedExternalList.size();
if(condition2 == true) i will merge bundles one and two,In the process
of real case test(Spring and Tomcat), i found that it's hardly to find
two bundles which have exactly the same usesExternal items, we should
also merge bundles who have *proportional* usesExternal items. In my
test case, it works good.
> And make sure you ignore all java.* to make the info smaller (in OSGi, java
> is always provided by the VM). Can you just print out something like
Yes, i have considered this situation, in the whole merge process, we
do not consider jdk supplies java.* classes.
>
> name usesInternal usesExternal
> g1 g2 a,b,c
> g2 d,e,f
>
Let's take example of Spring, this report is like this:
Name ---
usesInternal
org.springframework.jms2
---round1MergeBundle8,mergedBundle9,org.springframework.jca22,org.springframework.scheduling15,org.springframework.context14,round1MergeBundle5
org.springframework.jdbc11 ---
round1MergeBundle8,mergedBundle9
org.springframework.orm13
---
round1MergeBundle8,org.springframework.jdbc11,mergedBundle9,round1MergeBundle3
org.springframework.context14
--- round1MergeBundle8,round1MergeBundle1,mergedBundle9
org.springframework.scheduling15
---
round1MergeBundle8,mergedBundle9,org.springframework.context14,org.springframework.jdbc11
org.springframework.jca22
---
round1MergeBundle8,mergedBundle9,org.springframework.context14,org.springframework.scheduling15
round1MergeBundle1
--- round1MergeBundle8,mergedBundle9,org.springframework.metadata7
round1MergeBundle3
---
round1MergeBundle8,org.springframework.context14,round1MergeBundle5
round1MergeBundle4 ---
round1MergeBundle8,org.springframework.context14
round1MergeBundle5
---
round1MergeBundle8,round1MergeBundle3,org.springframework.context14,mergedBundle9
mergedBundle9
---
round1MergeBundle8,round1MergeBundle5,org.springframework.context14,org.springframework.metadata7
org.springframework.metadata7 --- none
round1MergeBundle8 --- none
As usesExternal element list is too long, i abridged them, you can get
the report details with usesExternal elements in attach file
"bundles_relation.txt", and get the bundles details in attach file
"SpringSplitTest.analyse"
> I hope we see one big bundle which is the core and then have to find rules to
> classify the remaining bundles. I expect there are the following categories:
>
> core implementation classes, lots of strongly connected packages
> api api classes, do not refer to core, very few imports
> bridge refer strongly to core and have expensive imports
>
>
> At this stage, the trick is to do some work by hand until you find you really
> understand the problem.
>
> It would be perfect if you could take a look at Tinkerpop and JUNG. I think
> it would be quite easy to visualize the graph of dependencies.
You mean my next step is developing a Tinkerpop or JUNG graphical view
for this report for user to adjust the bundles details manully ? Am i
right ? I will start learn Jung and start this job soon
>
> Kind regards,
>
> Peter Kriens
>
>
>
>
> On 26 jul 2011, at 16:24, Tiger Gui wrote:
>
>> Hi Peter,
>>
>> This is the whole application split algorithm here. After application
>> source code analyse algorithm described here[1], we can know each
>> package/class use which packages/classes and be used by which
>> packages/classes. Now, we just discuss package here, we treat package
>> as a single atom, each package has three important attributes, usedBy,
>> usesExternal and usesInternal, just like below:
>>
>> <package name="org.apache.catalina.deploy"
>> sources="/E:/GSoC/gsoc2011osgi/runtime-New_configuration/TomcatJava/bin"
>> size="30" usedBy="8" usesInternal="11" usesExternal="14" layer="6"
>> cycle="org.apache.catalina et al.">
>> <packageRef name="org.apache.catalina.core" type="usedBy"/>
>> <packageRef name="org.apache.catalina.startup" type="usedBy"/>
>> <packageRef name="org.apache.catalina.deploy" type="usedBy"/>
>> <packageRef name="org.apache.catalina.authenticator" type="usedBy"/>
>> <packageRef name="org.apache.catalina" type="usedBy"/>
>> <packageRef name="org.apache.catalina.mbeans" type="usedBy"/>
>> <packageRef name="org.apache.catalina.connector" type="usedBy"/>
>> <packageRef name="org.apache.catalina.realm" type="usedBy"/>
>> <packageRef name="java.lang" type="usesExternal"/>
>> <packageRef name="java.io" type="usesExternal"/>
>> <packageRef name="org.apache.catalina.deploy" type="usesInternal"/>
>> <packageRef name="org.apache.catalina.util" type="usesInternal"/>
>> <packageRef name="java.util" type="usesExternal"/>
>> <packageRef name="org.apache.juli.logging" type="usesInternal"/>
>> <packageRef name="org.apache.tomcat.util.res" type="usesInternal"/>
>> <packageRef name="java.beans" type="usesExternal"/>
>> <packageRef name="(default package)" type="usesExternal"/>
>> <packageRef name="org.apache.catalina" type="usesInternal"/>
>> <packageRef name="org.apache.catalina.mbeans" type="usesInternal"/>
>> <packageRef name="javax.management" type="usesExternal"/>
>> <packageRef name="namingResources" type="usesExternal"/>
>> <packageRef name="javax.naming" type="usesExternal"/>
>> <packageRef name="org.apache.naming" type="usesInternal"/>
>> <packageRef name="java.lang.reflect" type="usesExternal"/>
>> <packageRef name="javax.servlet" type="usesInternal"/>
>> <packageRef name="javax.servlet.annotation" type="usesInternal"/>
>> <packageRef name="org.apache.catalina.order" type="usesExternal"/>
>> <packageRef name="java.net" type="usesExternal"/>
>> <packageRef name="webXml" type="usesExternal"/>
>> <packageRef name="webXml.version" type="usesExternal"/>
>> <packageRef name="webxml" type="usesExternal"/>
>> <packageRef name="org.apache.catalina.core" type="usesInternal"/>
>> <packageRef name="javax.servlet.descriptor" type="usesInternal"/>
>> </package>
>>
>> usedBy means who use current package;
>> usesInternal means which packages current package use in current
>> application source code;
>> usesExternal means which packages current pakcage use not in current
>> application's source code.
>>
>> Now, we start split the whole application in to bundles, according to
>> above algorithm source code analyse algorithm, we can also know
>> package cycles in current application.
>>
>> 1. Treat each package cycle as a single bundle;
>> 2. Treat each packages not in cycle as a single bundle;
>> 3. Then we should decide which bundles can be merged into one new bundle;
>>
>> First round merge job:
>> 4. If one bundle's all usesInternal elements are in the other bundle,
>> these two bundles should be merged into a new bundle;
>>
>> Think about how to merge used by only bundle (bundle be used only by
>> other bundle, it does not rely on any other bundle):
>> Define two variable for used by only bundle:
>> sameUB: it means the number of two bundles have same usedBy elements.
>> sameUE: it menas the number of two bundles have same usedExternal
>> external package rely elements.
>>
>> boolean condition1 = 4 * sameUB >= one.usedByList.size() +
>> two.usedByList.size();
>> boolean condition2 = 2 * sameUE >= one.usedExternalList.size() +
>> two.usedExternalList.size();
>>
>> if condition1 or condition2 is true, we should merge these two usedBy
>> only bundle.
>>
>> Merge the other bundles:
>> 5.The core problem is how to decide two bundles(bundle one and bundle
>> two) can be merged or not.
>>
>> Define 5 variables:
>> uiNumber: the sum of bundle one's usesInternal elements in bundle two
>> number and bundle two's usesInternal elements in bundle one number;
>>
>> ubNumber:the sum of bundle one's usedBy elements in bundle two number
>> and bundle two's usedBy elements in bundle one number;
>>
>> sameUI: the same usesInternal number bundle one and two have
>>
>> sameUB: Be similar with used by only bundle's this variable
>>
>> sameUE: Be similar with used by only bundle's this variable
>>
>> Define these conditions:
>> boolean condition1 = 2 * uiNumber >= one.usesInternalList.size() +
>> two.usesInternalList.size();
>> boolean condition2 = 2 * ubNumber >= one.usedByList.size() +
>> two.usedByList.size() ;
>> boolean condition3 = 3.5 * sameUI >= one.usesInternalList.size() +
>> two.usesInternalList.size();
>> boolean condition4 = 4 * sameUB >= one.usedByList.size() +
>> two.usedByList.size();
>> boolean condition5 = 3 * sameUE >= one.usedExternalList.size() +
>> two.usedExternalList.size();
>>
>> If any above condition is true, these two bundles can be merged. But
>> these are another problem, if bundle A can be merged with B, but it
>> also can be merged with C, now, we should decide merge A with B or A
>> with C.
>>
>> Define the follow attribute:
>> int mergeFactor= 2 * (uiNumber + ubNumber) + 0.4 * (sameUI + sameUB) +
>> 0.2 * sameUE - number;
>>
>> If A and B's mergeFactor is 20 and A and C's mergeFactor is 30, we
>> should merge A and C.
>>
>> This is current merge algorithm in OSGiMaker, i will keep on improving
>> it, use class relationship factors or etc. You can find the source
>> code detail of this algorithm in class AnalyseJob of our project.
>>
>> The attach file is the analyse result document OSGiMaker analyse
>> Tomcat's source code and split it into bundles, you can have a review.
>>
>>
>> In fact, i did not want to bother you too much, but it seems that you
>> have enough time to help me to improve it, this is a good thing. If
>> you have any advises, please let me know, let's improving it together
>> :-)
>>
>> Thank you
>>
>> [1]
>> http://code.google.com/p/osgimaker/wiki/Implement_of_project_analyse_algorithm
>>
>> 2011/7/26 Peter Kriens <[email protected]>:
>>> Well, I do not think I am eager but I have a hard time getting a feeling
>>> where you are. You do not have to send reports, I want to see intermediate
>>> results and discuss issues if there are any. As I said earlier, it is not
>>> clear to me yet what the best algorithm is so that need to be worked out
>>> before we do the gui stuff.
>>>
>>> Kind regards,
>>>
>>> Peter Kriens
>>>
>>>
>>>
>>> On 26 jul 2011, at 11:24, Tiger Gui wrote:
>>>
>>>> In my schedule, i will report current status to you tomorrow as i
>>>> think i can get a usable version today, the whole analyse and split
>>>> algorithm is complex i have to organize a document to describe it
>>>> clearly. As you are really eager to see its progress, it is OK, i will
>>>> start the report now
>>>>
>>>> 2011/7/26 Peter Kriens <[email protected]>:
>>>>> If that is the case you have to provide more on going feedback. What
>>>>> happened to the analysis by hand?
>>>>>
>>>>> Kind regards,
>>>>>
>>>>> Peter Kriens
>>>>>
>>>>>
>>>>> On 26 jul 2011, at 11:02, Tiger Gui wrote:
>>>>>
>>>>>> No, Peter, i am really working hard for this project, you can check
>>>>>> the progress here[1]. Now, this tool can analyse a whole project and
>>>>>> export the its analyse result to bundles( I test it in Spring and
>>>>>> Tomcat project), if possible, you can install it in your Eclipse and
>>>>>> have a trial of it. But, i am still improving the split algorithm as
>>>>>> the current algorithm is not working perfect.
>>>>>>
>>>>>> I will supply a document about the current status of this project and
>>>>>> a simple guide for you to have a trial of it. I am really working very
>>>>>> hard for it these days :-(
>>>>>>
>>>>>> [1] http://code.google.com/p/osgimaker/updates/list
>>>>>>
>>>>>> 2011/7/26 Peter Kriens <[email protected]>:
>>>>>>> I am getting the feeling that you're not working very hard on this
>>>>>>> project and only does something just for the evaluations ...
>>>>>>>
>>>>>>> Peter Kriens
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards
>>>>>> ----------------------------------------------------
>>>>>> Tiger Gui [[email protected]]
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards
>>>> ----------------------------------------------------
>>>> Tiger Gui [[email protected]]
>>>
>>>
>>
>>
>>
>> --
>> Best Regards
>> ----------------------------------------------------
>> Tiger Gui [[email protected]]
>> <TomcatJava.analyse>
>
>
--
Best Regards
----------------------------------------------------
Tiger Gui [[email protected]]
Name ---
usesInternal ---
usesExternal
org.springframework.jms2 ---
round1MergeBundle8,mergedBundle9,org.springframework.jca22,org.springframework.scheduling15,org.springframework.context14,round1MergeBundle5,
---
javax.jms,org.apache.commons.logging,javax.naming,javax.resource.spi,javax.resource,javax.resource.spi.endpoint,org.apache.commons.pool,org.apache.commons.pool.impl,org.w3c.dom,org.aopalliance.intercept,
org.springframework.jdbc11 ---
round1MergeBundle8,mergedBundle9, ---
javax.sql,org.apache.commons.logging,javax.sql.rowset,com.sun.rowset,javax.xml.transform,javax.xml.transform.dom,org.w3c.dom,org.jboss.resource.adapter.jdbc,com.ibm.ws.rsadapter.jdbc,org.enhydra.jdbc.core,weblogic.jdbc.extensions,com.mchange.v2.c3p0,oracle.sql,javax.transaction,com.ibm.websphere.rsadapter,javax.naming,
org.springframework.orm13 ---
round1MergeBundle8,org.springframework.jdbc11,mergedBundle9,round1MergeBundle3,
---
com.ibatis.sqlmap.client,javax.sql,com.ibatis.sqlmap.engine.impl,com.ibatis.sqlmap.engine.transaction,org.apache.commons.logging,com.ibatis.sqlmap.client.event,com.ibatis.common.util,com.ibatis.sqlmap.engine.transaction.external,com.ibatis.sqlmap.engine.builder.xml,com.ibatis.common.xml,com.ibatis.sqlmap.engine.type,javax.servlet.http,javax.servlet,javax.jdo,oracle.toplink.queryframework,oracle.toplink.expressions,oracle.toplink.sessions,oracle.toplink.exceptions,sessions,oracle.toplink.internal.databaseaccess,oracle.toplink.jndi,oracle.toplink.tools.sessionmanagement,oracle.toplink.tools.sessionconfiguration,oracle.toplink.publicinterface,oracle.toplink.threetier,oracle.toplink.sessionbroker,org.aopalliance.intercept,oracle.toplink,oracle.toplink.logging,org.hibernate,org.hibernate.classic,org.hibernate.impl,org.hibernate.exception,org.hibernate.engine,org.hibernate.connection,javax.transaction,org.hibernate.type,org.hibernate.cache,org.hibernate.cfg,hibernate,hibernate.transaction,org.hibernate.transaction,hibernate.connection,hibernate.cache,org.hibernate.event,org.hibernate.dialect,org.hibernate.tool.hbm2ddl,org.hibernate.criterion,org.hibernate.transform,org.hibernate.jdbc,org.hibernate.context,org.hibernate.util,org.hibernate.event.def,org.hibernate.persister.entity,org.hibernate.usertype,javax.jdo.datastore,
org.springframework.context14 ---
round1MergeBundle8,round1MergeBundle1,mergedBundle9, ---
org.w3c.dom,org.springframework.jmx.export.annotation,org.springframework.context.annotation,org.springframework.context.weaving,org.springframework.beans.factory.aspectj,weblogic.management,com.ibm.websphere.management,org.apache.commons.logging,org.xml.sax,javax.naming,org.aopalliance.intercept,
org.springframework.scheduling15 ---
round1MergeBundle8,mergedBundle9,org.springframework.context14,org.springframework.jdbc11,
---
commonj.timers,javax.naming,org.apache.commons.logging,commonj.work,edu.emory.mathcs.backport.java.util.concurrent,org.quartz,org.quartz.spi,org.quartz.threadPool,javax.sql,org.quartz.impl,org.quartz.scheduler.classLoadHelper,org.quartz.simpl,org.quartz.jobStore,org.quartz.scheduler,org.quartz.impl.jdbcjobstore,org.quartz.utils,org.quartz.xml,
org.springframework.jca22 ---
round1MergeBundle8,mergedBundle9,org.springframework.context14,org.springframework.scheduling15,
---
javax.resource.spi,javax.resource,javax.resource.spi.work,javax.resource.cci,org.apache.commons.logging,javax.naming,javax.resource.spi.endpoint,javax.transaction.xa,org.aopalliance.intercept,org.aopalliance.aop,javax.transaction,com.sun.enterprise.connectors.work,org.jboss.resource.work,javax.management,
round1MergeBundle1 ---
round1MergeBundle8,mergedBundle9,org.springframework.metadata7, ---
javax.management,javax.management.modelmbean,org.aopalliance.intercept,org.apache.commons.logging,javax.management.remote,javax.management.openmbean,weblogic.management,javax.naming,com.ibm.websphere.management,javax.mail.internet,javax.mail,javax.activation,mime,
round1MergeBundle3 ---
round1MergeBundle8,org.springframework.context14,round1MergeBundle5, ---
org.apache.commons.logging,freemarker.cache,freemarker.template,net.sf.jasperreports.engine,net.sf.jasperreports.engine.data,net.sf.jasperreports.engine.export,org.apache.velocity.runtime.resource.loader,spring.resource.loader,spring.resource,org.apache.commons.collections,org.apache.velocity.runtime,org.apache.velocity.exception,org.apache.velocity.runtime.resource,org.apache.velocity.app,runtime.log,resource,file.resource.loader,org.apache.velocity,org.apache.velocity.context,org.apache.velocity.runtime.log,javax.servlet,javax.servlet.http,javax.servlet.jsp,javax.servlet.jsp.tagext,org.springframework.web.servlet.tags.form.SelectTag,org.xml.sax,javax.servlet.jsp.jstl.core,org.apache.struts.tiles,org.apache.struts.tiles.xmlDefinition,javax.sql,net.sf.jasperreports.engine.design,net.sf.jasperreports.engine.util,net.sf.jasperreports.engine.xml,org.apache.velocity.app.tools,org.apache.velocity.tools.generic,springMacro.resource.loader,velocimacro,layout,org.apache.velocity.tools.view.context,org.apache.velocity.tools.view.servlet,org.apache.velocity.tools.view,org.apache.velocity.tools.view.tools,org.apache.tiles.preparer,org.apache.tiles,org.apache.tiles.factory,org.apache.tiles.context,org.apache.tiles.definition,org.apache.tiles.context.ChainTilesContextFactory,org.apache.tiles.servlet.context,org.apache.tiles.jsp.context,org.apache.tiles.locale,org.apache.tiles.impl.BasicTilesContainer,org.apache.tiles.definition.digester.DigesterDefinitionsReader,org.apache.tiles.factory.TilesContainerFactory,org.apache.tiles.access,org.apache.tiles.web.util,org.apache.tiles.locale.impl,org.apache.poi.hssf.usermodel,org.apache.poi.poifs.filesystem,com.lowagie.text,com.lowagie.text.pdf,jxl,jxl.write,javax.xml.transform,org.w3c.dom,javax.xml.transform.dom,javax.xml.transform.stream,freemarker.ext.jsp,freemarker.ext.servlet,freemarker.core,org.apache.log4j,javax.portlet,user.login,user,DispatcherPortlet,org.apache.commons.fileupload.portlet,org.apache.commons.fileupload,org.apache.commons.fileupload.disk,javax.servlet.context,spring,org.apache.struts.action,org.apache.struts.config,org.apache.struts.actions,org.apache.struts.util,org.apache.commons.beanutils,javax.servlet.error,org.apache.commons.attributes,javax.faces.el,javax.faces.context,javax.faces.event,javax.faces.application,org.apache.commons.fileupload.servlet,DispatcherServlet,javax.servlet.jsp.jstl.fmt,HtmlCharacterEntityReferences,javax.servlet.jsp.el,org.apache.taglibs.standard.lang.support,javax.servlet.include,javax.servlet.forward,webapp,ContextLoader,
round1MergeBundle4 ---
round1MergeBundle8,org.springframework.context14, ---
groovy.lang,org.codehaus.groovy.control,bsh,org.jruby.exceptions,org.jruby,org.jruby.runtime.builtin,org.jruby.runtime,org.jruby.ast,org.jruby.javasupport,org.apache.commons.logging,net.sf.cglib.proxy,net.sf.cglib.core,net.sf.cglib.asm,org.aopalliance.aop,org.w3c.dom,net.sf.ehcache,net.sf.ehcache.store,net.sf.ehcache.constructs.blocking,net.sf.ehcache.event,net.sf.ehcache.bootstrap,
round1MergeBundle5 ---
round1MergeBundle8,round1MergeBundle3,org.springframework.context14,mergedBundle9,
---
javax.xml.rpc.server,org.apache.commons.logging,javax.xml.rpc,javax.servlet,javax.xml.namespace,javax.xml.rpc.soap,org.aopalliance.intercept,javax.xml.rpc.security.auth,javax.xml.rpc.service.endpoint,javax.xml.rpc.session,javax.xml.rpc.encoding,org.apache.axis.encoding.ser,org.aopalliance.aop,com.caucho.hessian.server,com.caucho.hessian.io,com.caucho.hessian.client,javax.servlet.http,com.caucho.burlap.client,com.caucho.burlap.server,com.caucho.burlap.io,javax.naming,javax.rmi,org.omg.CORBA,com.evermind.server.rmi,org.apache.commons.httpclient,org.apache.commons.httpclient.params,org.apache.commons.httpclient.methods,
mergedBundle9 ---
round1MergeBundle8,round1MergeBundle5,org.springframework.context14,org.springframework.metadata7,
---
org.w3c.dom,org.aopalliance.intercept,javax.ejb,javax.naming,javax.rmi,org.apache.commons.logging,javax.jms,org.aopalliance.aop,oracle.j2ee.transaction,com.evermind.server,javax.transaction,com.ibm.ws.Transaction,com.ibm.wsspi.uow,weblogic.transaction,org.objectweb.jotm,org.springframework.transaction.annotation,org.springframework.transaction.aspectj,
org.springframework.metadata7 --- ---
org.apache.commons.attributes,
round1MergeBundle8 --- ---
java,org.apache.commons.logging,org.objectweb.asm,org.objectweb.asm.commons,org.apache.commons.collections.map,edu.emory.mathcs.backport.java.util.concurrent,org.apache.commons.collections,org.eclipse.core.runtime,org.apache.log4j.xml,org.apache.log4j,javax.xml.transform,org.w3c.dom,org.xml.sax,javax.xml.parsers,net.sf.cglib.proxy,org.springframework.beans.factory.annotation,ClassFilter,org.aopalliance.intercept,org.aopalliance.aop,Pointcut,MethodMatcher,org.aspectj.lang,org.aspectj.weaver.tools,org.aspectj.lang.reflect,org.aspectj.runtime.internal,org.aspectj.weaver.ast,org.aspectj.weaver.reflect,org.aspectj.weaver,org.aspectj.weaver.internal.tools,org.aspectj.bridge,org.aspectj.weaver.patterns,org.springframework.beans.factory.aspectj,org.springframework.aop.aspectj.annotation,net.sf.cglib.transform.impl,net.sf.cglib.core,org.aspectj.util,com.jamonapi,org.apache.commons.pool,org.apache.commons.pool.impl,
