Hi Peter,
To visualize things, i have a simple review to JUNG, it seems that it
is good solution for display graph or tree in a canvas, but it is a
problem that JUNG is developed for AWT/SWING, not suitable for Eclipse
SWT/JFace environment, so, i will continue searching for the best
solution, may be we can even consider Eclipse GMF technology
To report things:
I have changed the report to a HTML file in the attach file, it's
still Spring's OSGi split report, it looks much better :-)
2011/7/29 Peter Kriens <peter.kri...@aqute.biz>:
> Rule #1, as long as I see you're engaged you do not have to worry about the
> evaluation. So do not do things for my sake, do them because you believe it
> is the best way forward. I think Tinkerpop and JUNG are a great way to
> visualize, but do not take my word. I have the greatest respect for people
> who do not listen to me because they are convinced they know better. That out
> of the way.
>
> This whole things is very much about understanding and our visual brains is
> by far the most powerful tool on earth to understand things. And you're not
> taking advantage of it :-) We are looking for structure here, the names are
> only relevant later. So to see structure, tables are perfect. Which means you
> have to have short names. Just using the last part of a package
> (com.springframework.jca, take jca). I.e. a name like round1MergeBundle8 is
> ok to call b8. You can always later provide a list of translations. This is
> also the reason why we need to use packages, classes are too many off.
> Visualization is everything in this phase because we need to find the rules
> of the structure.
>
> So lets retry. First give me a list of all groups (named g1 .. gn) and their
> package content
>
>
> Group Contains Uses Used By
> External
> g3 com.springframework.{jca,jdbc,...}, g4, g5, g7 g2
> org.w3.xml, javax.persistence,
> ...
> com.oracle.g11.{impl,whatever}
>
> It might be easiest to generate HTML so you can use tables. Don't make it
> look fancy, just use the tools to get insight in the structure:
>
> <html>
> <body>
> <table>
> <tr>
> <th>Group</th>
> <th>Contains</th>
> <th>Uses</th>
> <th>Used By</th>
> <th>External</th>
> </tr>
> <tr>
> <td>g3</td>
> <td>com.springframework.{jca,jdbc,...},...</td>
> <td>g4, g5, g7</td>
> <td>g2</td>
> <td>org.w3.xml,
> javax.persistence,com.oracle.g11.{impl,whatever}</td>
> </tr>
> </table>
> </body>
> </html>
>
> Shortening the names is everything or else your eyes drown in a sea of
> characters.
>
> Kind regards,
>
> Peter Kriens
>
>
>
> On 29 jul 2011, at 05:16, Tiger Gui wrote:
>
>> Hi Peter,
>>
>> Thank you for your reply first, please see the following comments
>>
>> 2011/7/28 Peter Kriens <peter.kri...@aqute.biz>:
>>> It is good that you use a real project now. What I like to see how many
>>> "bundles" we have after the first round, where you group strongly connected
>>> packages. This should already simplify because the nr of entities will be
>>> smaller.
>>>
>>> I need to see a real case. Step 1 is ok, but maybe you can add merging of
>>> bundles that have identical usesExternal (imports).
>>
>> Yes, I do merge the bundles that have identical usesExternal
>> (imports), i described in the former mail,
>>
>> We define sameUE: it menas the number of two bundles have same
>> usedExternal external package rely elements.
>>
>> boolean condition2 = 3 * sameUE >= one.usedExternalList.size() +
>> two.usedExternalList.size();
>>
>> if(condition2 == true) i will merge bundles one and two,In the process
>> of real case test(Spring and Tomcat), i found that it's hardly to find
>> two bundles which have exactly the same usesExternal items, we should
>> also merge bundles who have *proportional* usesExternal items. In my
>> test case, it works good.
>>
>>> And make sure you ignore all java.* to make the info smaller (in OSGi, java
>>> is always provided by the VM). Can you just print out something like
>>
>> Yes, i have considered this situation, in the whole merge process, we
>> do not consider jdk supplies java.* classes.
>>
>>>
>>> name usesInternal usesExternal
>>> g1 g2 a,b,c
>>> g2 d,e,f
>>>
>>
>> Let's take example of Spring, this report is like this:
>> Name ---
>> usesInternal
>> org.springframework.jms2
>> ---round1MergeBundle8,mergedBundle9,org.springframework.jca22,org.springframework.scheduling15,org.springframework.context14,round1MergeBundle5
>> org.springframework.jdbc11 ---
>> round1MergeBundle8,mergedBundle9
>> org.springframework.orm13
>> ---
>> round1MergeBundle8,org.springframework.jdbc11,mergedBundle9,round1MergeBundle3
>> org.springframework.context14
>> --- round1MergeBundle8,round1MergeBundle1,mergedBundle9
>> org.springframework.scheduling15
>> ---
>> round1MergeBundle8,mergedBundle9,org.springframework.context14,org.springframework.jdbc11
>> org.springframework.jca22
>> ---
>> round1MergeBundle8,mergedBundle9,org.springframework.context14,org.springframework.scheduling15
>> round1MergeBundle1
>> --- round1MergeBundle8,mergedBundle9,org.springframework.metadata7
>> round1MergeBundle3
>> ---
>> round1MergeBundle8,org.springframework.context14,round1MergeBundle5
>> round1MergeBundle4 ---
>> round1MergeBundle8,org.springframework.context14
>> round1MergeBundle5
>> ---
>> round1MergeBundle8,round1MergeBundle3,org.springframework.context14,mergedBundle9
>> mergedBundle9
>> ---
>> round1MergeBundle8,round1MergeBundle5,org.springframework.context14,org.springframework.metadata7
>> org.springframework.metadata7 --- none
>> round1MergeBundle8 --- none
>>
>> As usesExternal element list is too long, i abridged them, you can get
>> the report details with usesExternal elements in attach file
>> "bundles_relation.txt", and get the bundles details in attach file
>> "SpringSplitTest.analyse"
>>
>>
>>> I hope we see one big bundle which is the core and then have to find rules
>>> to classify the remaining bundles. I expect there are the following
>>> categories:
>>>
>>> core implementation classes, lots of strongly connected packages
>>> api api classes, do not refer to core, very few imports
>>> bridge refer strongly to core and have expensive imports
>>>
>>>
>>> At this stage, the trick is to do some work by hand until you find you
>>> really understand the problem.
>>>
>>> It would be perfect if you could take a look at Tinkerpop and JUNG. I think
>>> it would be quite easy to visualize the graph of dependencies.
>>
>> You mean my next step is developing a Tinkerpop or JUNG graphical view
>> for this report for user to adjust the bundles details manully ? Am i
>> right ? I will start learn Jung and start this job soon
>>
>>>
>>> Kind regards,
>>>
>>> Peter Kriens
>>>
>>>
>>>
>>>
>>> On 26 jul 2011, at 16:24, Tiger Gui wrote:
>>>
>>>> Hi Peter,
>>>>
>>>> This is the whole application split algorithm here. After application
>>>> source code analyse algorithm described here[1], we can know each
>>>> package/class use which packages/classes and be used by which
>>>> packages/classes. Now, we just discuss package here, we treat package
>>>> as a single atom, each package has three important attributes, usedBy,
>>>> usesExternal and usesInternal, just like below:
>>>>
>>>> <package name="org.apache.catalina.deploy"
>>>> sources="/E:/GSoC/gsoc2011osgi/runtime-New_configuration/TomcatJava/bin"
>>>> size="30" usedBy="8" usesInternal="11" usesExternal="14" layer="6"
>>>> cycle="org.apache.catalina et al.">
>>>> <packageRef name="org.apache.catalina.core" type="usedBy"/>
>>>> <packageRef name="org.apache.catalina.startup" type="usedBy"/>
>>>> <packageRef name="org.apache.catalina.deploy" type="usedBy"/>
>>>> <packageRef name="org.apache.catalina.authenticator" type="usedBy"/>
>>>> <packageRef name="org.apache.catalina" type="usedBy"/>
>>>> <packageRef name="org.apache.catalina.mbeans" type="usedBy"/>
>>>> <packageRef name="org.apache.catalina.connector" type="usedBy"/>
>>>> <packageRef name="org.apache.catalina.realm" type="usedBy"/>
>>>> <packageRef name="java.lang" type="usesExternal"/>
>>>> <packageRef name="java.io" type="usesExternal"/>
>>>> <packageRef name="org.apache.catalina.deploy" type="usesInternal"/>
>>>> <packageRef name="org.apache.catalina.util" type="usesInternal"/>
>>>> <packageRef name="java.util" type="usesExternal"/>
>>>> <packageRef name="org.apache.juli.logging" type="usesInternal"/>
>>>> <packageRef name="org.apache.tomcat.util.res" type="usesInternal"/>
>>>> <packageRef name="java.beans" type="usesExternal"/>
>>>> <packageRef name="(default package)" type="usesExternal"/>
>>>> <packageRef name="org.apache.catalina" type="usesInternal"/>
>>>> <packageRef name="org.apache.catalina.mbeans" type="usesInternal"/>
>>>> <packageRef name="javax.management" type="usesExternal"/>
>>>> <packageRef name="namingResources" type="usesExternal"/>
>>>> <packageRef name="javax.naming" type="usesExternal"/>
>>>> <packageRef name="org.apache.naming" type="usesInternal"/>
>>>> <packageRef name="java.lang.reflect" type="usesExternal"/>
>>>> <packageRef name="javax.servlet" type="usesInternal"/>
>>>> <packageRef name="javax.servlet.annotation" type="usesInternal"/>
>>>> <packageRef name="org.apache.catalina.order" type="usesExternal"/>
>>>> <packageRef name="java.net" type="usesExternal"/>
>>>> <packageRef name="webXml" type="usesExternal"/>
>>>> <packageRef name="webXml.version" type="usesExternal"/>
>>>> <packageRef name="webxml" type="usesExternal"/>
>>>> <packageRef name="org.apache.catalina.core" type="usesInternal"/>
>>>> <packageRef name="javax.servlet.descriptor" type="usesInternal"/>
>>>> </package>
>>>>
>>>> usedBy means who use current package;
>>>> usesInternal means which packages current package use in current
>>>> application source code;
>>>> usesExternal means which packages current pakcage use not in current
>>>> application's source code.
>>>>
>>>> Now, we start split the whole application in to bundles, according to
>>>> above algorithm source code analyse algorithm, we can also know
>>>> package cycles in current application.
>>>>
>>>> 1. Treat each package cycle as a single bundle;
>>>> 2. Treat each packages not in cycle as a single bundle;
>>>> 3. Then we should decide which bundles can be merged into one new bundle;
>>>>
>>>> First round merge job:
>>>> 4. If one bundle's all usesInternal elements are in the other bundle,
>>>> these two bundles should be merged into a new bundle;
>>>>
>>>> Think about how to merge used by only bundle (bundle be used only by
>>>> other bundle, it does not rely on any other bundle):
>>>> Define two variable for used by only bundle:
>>>> sameUB: it means the number of two bundles have same usedBy elements.
>>>> sameUE: it menas the number of two bundles have same usedExternal
>>>> external package rely elements.
>>>>
>>>> boolean condition1 = 4 * sameUB >= one.usedByList.size() +
>>>> two.usedByList.size();
>>>> boolean condition2 = 2 * sameUE >= one.usedExternalList.size() +
>>>> two.usedExternalList.size();
>>>>
>>>> if condition1 or condition2 is true, we should merge these two usedBy
>>>> only bundle.
>>>>
>>>> Merge the other bundles:
>>>> 5.The core problem is how to decide two bundles(bundle one and bundle
>>>> two) can be merged or not.
>>>>
>>>> Define 5 variables:
>>>> uiNumber: the sum of bundle one's usesInternal elements in bundle two
>>>> number and bundle two's usesInternal elements in bundle one number;
>>>>
>>>> ubNumber:the sum of bundle one's usedBy elements in bundle two number
>>>> and bundle two's usedBy elements in bundle one number;
>>>>
>>>> sameUI: the same usesInternal number bundle one and two have
>>>>
>>>> sameUB: Be similar with used by only bundle's this variable
>>>>
>>>> sameUE: Be similar with used by only bundle's this variable
>>>>
>>>> Define these conditions:
>>>> boolean condition1 = 2 * uiNumber >= one.usesInternalList.size() +
>>>> two.usesInternalList.size();
>>>> boolean condition2 = 2 * ubNumber >= one.usedByList.size() +
>>>> two.usedByList.size() ;
>>>> boolean condition3 = 3.5 * sameUI >= one.usesInternalList.size() +
>>>> two.usesInternalList.size();
>>>> boolean condition4 = 4 * sameUB >= one.usedByList.size() +
>>>> two.usedByList.size();
>>>> boolean condition5 = 3 * sameUE >= one.usedExternalList.size() +
>>>> two.usedExternalList.size();
>>>>
>>>> If any above condition is true, these two bundles can be merged. But
>>>> these are another problem, if bundle A can be merged with B, but it
>>>> also can be merged with C, now, we should decide merge A with B or A
>>>> with C.
>>>>
>>>> Define the follow attribute:
>>>> int mergeFactor= 2 * (uiNumber + ubNumber) + 0.4 * (sameUI + sameUB) +
>>>> 0.2 * sameUE - number;
>>>>
>>>> If A and B's mergeFactor is 20 and A and C's mergeFactor is 30, we
>>>> should merge A and C.
>>>>
>>>> This is current merge algorithm in OSGiMaker, i will keep on improving
>>>> it, use class relationship factors or etc. You can find the source
>>>> code detail of this algorithm in class AnalyseJob of our project.
>>>>
>>>> The attach file is the analyse result document OSGiMaker analyse
>>>> Tomcat's source code and split it into bundles, you can have a review.
>>>>
>>>>
>>>> In fact, i did not want to bother you too much, but it seems that you
>>>> have enough time to help me to improve it, this is a good thing. If
>>>> you have any advises, please let me know, let's improving it together
>>>> :-)
>>>>
>>>> Thank you
>>>>
>>>> [1]
>>>> http://code.google.com/p/osgimaker/wiki/Implement_of_project_analyse_algorithm
>>>>
>>>> 2011/7/26 Peter Kriens <peter.kri...@aqute.biz>:
>>>>> Well, I do not think I am eager but I have a hard time getting a feeling
>>>>> where you are. You do not have to send reports, I want to see
>>>>> intermediate results and discuss issues if there are any. As I said
>>>>> earlier, it is not clear to me yet what the best algorithm is so that
>>>>> need to be worked out before we do the gui stuff.
>>>>>
>>>>> Kind regards,
>>>>>
>>>>> Peter Kriens
>>>>>
>>>>>
>>>>>
>>>>> On 26 jul 2011, at 11:24, Tiger Gui wrote:
>>>>>
>>>>>> In my schedule, i will report current status to you tomorrow as i
>>>>>> think i can get a usable version today, the whole analyse and split
>>>>>> algorithm is complex i have to organize a document to describe it
>>>>>> clearly. As you are really eager to see its progress, it is OK, i will
>>>>>> start the report now
>>>>>>
>>>>>> 2011/7/26 Peter Kriens <peter.kri...@aqute.biz>:
>>>>>>> If that is the case you have to provide more on going feedback. What
>>>>>>> happened to the analysis by hand?
>>>>>>>
>>>>>>> Kind regards,
>>>>>>>
>>>>>>> Peter Kriens
>>>>>>>
>>>>>>>
>>>>>>> On 26 jul 2011, at 11:02, Tiger Gui wrote:
>>>>>>>
>>>>>>>> No, Peter, i am really working hard for this project, you can check
>>>>>>>> the progress here[1]. Now, this tool can analyse a whole project and
>>>>>>>> export the its analyse result to bundles( I test it in Spring and
>>>>>>>> Tomcat project), if possible, you can install it in your Eclipse and
>>>>>>>> have a trial of it. But, i am still improving the split algorithm as
>>>>>>>> the current algorithm is not working perfect.
>>>>>>>>
>>>>>>>> I will supply a document about the current status of this project and
>>>>>>>> a simple guide for you to have a trial of it. I am really working very
>>>>>>>> hard for it these days :-(
>>>>>>>>
>>>>>>>> [1] http://code.google.com/p/osgimaker/updates/list
>>>>>>>>
>>>>>>>> 2011/7/26 Peter Kriens <peter.kri...@aqute.biz>:
>>>>>>>>> I am getting the feeling that you're not working very hard on this
>>>>>>>>> project and only does something just for the evaluations ...
>>>>>>>>>
>>>>>>>>> Peter Kriens
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards
>>>>>>>> ----------------------------------------------------
>>>>>>>> Tiger Gui [tigergui1...@gmail.com]
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards
>>>>>> ----------------------------------------------------
>>>>>> Tiger Gui [tigergui1...@gmail.com]
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards
>>>> ----------------------------------------------------
>>>> Tiger Gui [tigergui1...@gmail.com]
>>>> <TomcatJava.analyse>
>>>
>>>
>>
>>
>>
>> --
>> Best Regards
>> ----------------------------------------------------
>> Tiger Gui [tigergui1...@gmail.com]
>> <bundles_relation.txt><SpringSplitTest.analyse>
>
>
--
Best Regards
----------------------------------------------------
Tiger Gui [tigergui1...@gmail.com]
