Re: Feedback

[Richard S. Hall]

This looks interesting so far...

-> richard

On 8/11/11 10:39 PM, Tiger Gui wrote:

Hi,

You guys can find the attach file here[1], but if you are really
interested in this project, i suggest that you have a trial of
OSGiMaker, its update center is [2], you should install Bndtools
first, and then install OSGiMaker, you can get relative documents
here[3].

[1] http://osgimaker.googlecode.com/svn/trunk/OSGiMakerDemo/
[2] http://osgimaker.googlecode.com/svn/trunk/OSGiMakerUpdateCenter/
[3] 
http://code.google.com/p/osgimaker/wiki/Implement_of_project_analyse_algorithm

2011/8/11 Nicolas Lalevée<nicolas.lale...@hibnet.org>:

If I may, could put the attached documents somewhere we could download and see 
them, they get filtered throw the mailing list. This topic seems very 
interesting, all your discussion seems too, reading Peter being enthusiast; all 
this teasing without getting the actual content, so frustrating ! :)

cheers,
Nicolas

Le 11 août 2011 à 08:14, Tiger Gui a écrit :

Ok, i will keep on improving the visualization things and improve the
split algorithm according to the visualization result :-)

2011/8/10 Peter Kriens<peter.kri...@aqute.biz>:

Make sure you get the naming + layout right on these examples before you take 
other products. This is the stage where you really want to focus on the tool. 
You now get more understanding of the problem because you now know the 
structure. So I really think we now should do:

        - better visualization of the contents (naming mostly, maybe show the 
inside package names when zoomed)
        - show the imports somehow

It is not that useful to now analyze a lot of projects, it is understanding the 
structure of the projects that counts. These lessons must then be applied to 
other projects, which will give new problems, solve them, ad nauseum.

But this is really starting to look interestingly.

Kind regards,

        Peter Kriens


On 10 aug 2011, at 08:29, Tiger Gui wrote:

And this is Spring's packages relation figure and bundles relationship
figure in the attach png files. And you can get the bundle details in
the html split report generated by OSGiMaker.

Now, we use Jung to read the visualize the split result of OSGiMaker,
in the next few days, i will integrate the visualize tool with
OSGiMaker, help users to check the bundles relation in the Eclipse
once he use OSGiMaker to split the application.

2011/8/9 Peter Kriens<peter.kri...@aqute.biz>:

This is starting to look really interesting!

Can you make your own layout? Would be nice to sort the "bundles" in layers. 
The core should be at the bottom and when you go up, you should see the higher layers 
depend on the lower layers. This is never perfect but you cannot have cycles by 
definition, they've been removed earlier.

It is important to minimize the number of imports so they need also be grouped 
in such a way that you have a minimum set of import groups. They should be 
visible on the edge of the picture

Also spend some time on shortening the names without loosing the recognition.

Good work! As I said, visualizing this stuff is all important.

Kind regards,

        Peter Kriens


On 9 aug 2011, at 08:43, Tiger Gui wrote:

Hi Peter,

This is Tomcat's bundles relationship figure, i use Jung to  visualize
the dependencies, arrow from A to B means bundle A need bundle B. And
the html attach file is the bundles relationship report

2011/8/2 Tiger Gui<tigergui1...@gmail.com>:

Yeah, the imports are too much, but there is not any problem.

The good new is that i am working for the visualize job of  the
dependencies, it will give us a more clear graph about the bundles
relationship,  just wait for 2 -3 days, I am on my way of playing with
JUNG

2011/8/2 Peter Kriens<peter.kri...@aqute.biz>:

Can you take a look at the imports? There seems to be many class names?

I also think you should try to shorten the package names. Maybe names like 
oac.comet (org.apache.catalina.comet) would already help.

And we really need visualize the dependencies ... This is however a great start.

Kind regards,

        Peter Kriens

On 30 jul 2011, at 16:02, Tiger Gui wrote:

Hi Peter,

I changed the merge situation a little,

FROM:
boolean condition4 = 4 * sameUB>= one.usedByList.size() +
two.usedByList.size();
TO:
boolean condition4 = 3 * sameUB>= one.usedByList.size() +
two.usedByList.size();

Try the analyse and split algorithm in Tomcat application again, and
we got another report, in the attach file. It seems that this bundle
structure is more reasonable

2011/7/30 Tiger Gui<tigergui1...@gmail.com>:

Hi Peter,

I have generated the report for Tomcat too, in the attach file, and
improve the report as you wish, just see the comments

2011/7/29 Peter Kriens<peter.kri...@aqute.biz>:

Now we're getting somewhere! Could you:

1) Please rename names like round1MergeBundle8 to just R18 or so ...

I have renamed round1MergeBundle8 to just R1B8, BTW, you should know
that users can change the bundle name as he wish in our tools,
OSGiMaker has supplies the feature already, we have a GUI bundle edit
editor

2) The imports look like they sometimes contain classes? And it would be nice if you 
put a</br>  after each one of them (maybe combine the ones with the same 
prefix?)

No, the imports are all package, i add a "<br />" every 5 imports, if
we add a "<br/>" after each of them, it will be a long list, not very
comfortable to read

3) Can you sort on the number of contains entries? I.e. biggest bundle first

Yeah, i have sorted it already, the biggest first

4) Could you link the bundles with an<a href="..">R18</a>?

Yeah, i have add the http link already in the report

5) Could you make the names of the

???

6) Get rid of the .*?

It is easy to get rid of the .*, but i think it make things more
clear, make us know, the bundles includes all the classes in the
packages

7) Also generate one of these for catalina/Tomcat?

Actually, this spring one is pretty neat.

I am a bit curious about the jdbc stuff. I assume all the packages in 
org.springframework.jdbc11 are strongly connected? What are the packages that 
refer to com.ibm.websphere.rsadapter and com.mchange.v2.c3p0, etc? These are 
the VERY expensive dependencies and I wonder how they get merged in so badly in 
this bundle.

Similar to the orm bundle, who is the culprit (class?) that points to 
hibernate? You see why this is bad? In a good design the 
org.springframework.orm should not be strongly connected to something like 
hibernate. The hibernate should be a bridge package but not have internal 
references.
Interesting! Kind regards,

        Peter Kriens




On 29 jul 2011, at 11:32, Tiger Gui wrote:

Hi Peter,

To visualize things, i have a simple review to JUNG, it seems that it
is good solution for display graph or tree in a canvas, but it is a
problem that JUNG is developed for AWT/SWING, not suitable for Eclipse
SWT/JFace environment, so, i will continue searching for the best
solution, may be we can even consider Eclipse GMF technology


To report things:
I have changed the report to a HTML file in the attach file, it's
still Spring's OSGi split report, it looks much better :-)

2011/7/29 Peter Kriens<peter.kri...@aqute.biz>:

Rule #1, as long as I see you're engaged you do not have to worry about the 
evaluation. So do not do things for my sake, do them because you believe it is 
the best way forward. I think Tinkerpop and JUNG are a great way to visualize, 
but do not take my word. I have the greatest respect for people who do not 
listen to me because they are convinced they know better. That out of the way.

This whole things is very much about understanding and our visual brains is by 
far the most powerful tool on earth to understand things. And you're not taking 
advantage of it :-) We are looking for structure here, the names are only 
relevant later. So to see structure, tables are perfect. Which means you have 
to have short names. Just using the last part of a package 
(com.springframework.jca, take jca). I.e. a name like round1MergeBundle8 is ok 
to call b8. You can always later provide a list of translations. This is also 
the reason why we need to use packages, classes are too many off. Visualization 
is everything in this phase because we need to find the rules of the structure.

So lets retry. First give me a list of all groups (named g1 .. gn) and their 
package content


Group  Contains                              Uses           Used By      
External
g3     com.springframework.{jca,jdbc,...},   g4, g5, g7     g2           
org.w3.xml, javax.persistence,
       ...                                                               
com.oracle.g11.{impl,whatever}

It might be easiest to generate HTML so you can use tables. Don't make it look 
fancy, just use the tools to get insight in the structure:

<html>
   <body>
       <table>
          <tr>
             <th>Group</th>
             <th>Contains</th>
             <th>Uses</th>
             <th>Used By</th>
             <th>External</th>
          </tr>
          <tr>
            <td>g3</td>
            <td>com.springframework.{jca,jdbc,...},...</td>
            <td>g4, g5, g7</td>
            <td>g2</td>
            <td>org.w3.xml, 
javax.persistence,com.oracle.g11.{impl,whatever}</td>
          </tr>
        </table>
    </body>
</html>

Shortening the names is everything or else your eyes drown in a sea of 
characters.

Kind regards,

        Peter Kriens



On 29 jul 2011, at 05:16, Tiger Gui wrote:

Hi Peter,

Thank you for your reply first, please see the following comments

2011/7/28 Peter Kriens<peter.kri...@aqute.biz>:

It is good that you use a real project now. What I like to see how many 
"bundles" we have after the first round, where you group strongly connected 
packages. This should already simplify because the nr of entities will be smaller.

I need to see a real case. Step 1 is ok, but maybe you can add merging of 
bundles that have identical usesExternal (imports).

Yes, I do merge the bundles that have identical usesExternal
(imports), i described in the former mail,

We define sameUE: it menas the number of two bundles have same
usedExternal external package rely elements.

boolean condition2 =  3 * sameUE>= one.usedExternalList.size() +
two.usedExternalList.size();

if(condition2 == true) i will merge bundles one and two,In the process
of real case test(Spring and Tomcat), i found that it's hardly to find
two bundles which have exactly the same usesExternal items, we should
also merge bundles who have  *proportional* usesExternal items. In my
test case, it works good.

And make sure you ignore all java.* to make the info smaller (in OSGi, java is 
always provided by the VM). Can you just print out something like

Yes, i have considered this situation, in the whole merge process, we
do not consider jdk supplies java.* classes.

name            usesInternal            usesExternal
g1              g2                      a,b,c
g2                                      d,e,f

Let's take example of Spring, this report is like this:
Name                          ---                                             
usesInternal
org.springframework.jms2
---round1MergeBundle8,mergedBundle9,org.springframework.jca22,org.springframework.scheduling15,org.springframework.context14,round1MergeBundle5
org.springframework.jdbc11     ---            round1MergeBundle8,mergedBundle9
org.springframework.orm13
---           
round1MergeBundle8,org.springframework.jdbc11,mergedBundle9,round1MergeBundle3
org.springframework.context14
---           round1MergeBundle8,round1MergeBundle1,mergedBundle9
org.springframework.scheduling15
---           
round1MergeBundle8,mergedBundle9,org.springframework.context14,org.springframework.jdbc11
org.springframework.jca22
---           
round1MergeBundle8,mergedBundle9,org.springframework.context14,org.springframework.scheduling15
round1MergeBundle1
---           round1MergeBundle8,mergedBundle9,org.springframework.metadata7
round1MergeBundle3
---           
round1MergeBundle8,org.springframework.context14,round1MergeBundle5
round1MergeBundle4     ---            
round1MergeBundle8,org.springframework.context14
round1MergeBundle5
---           
round1MergeBundle8,round1MergeBundle3,org.springframework.context14,mergedBundle9
mergedBundle9
---           
round1MergeBundle8,round1MergeBundle5,org.springframework.context14,org.springframework.metadata7
org.springframework.metadata7     ---         none
round1MergeBundle8     ---                    none

As usesExternal element list is too long, i abridged them, you can get
the report details with usesExternal elements in attach file
"bundles_relation.txt", and get the bundles details in attach file
"SpringSplitTest.analyse"

I hope we see one big bundle which is the core and then have to find rules to 
classify the remaining bundles. I expect there are the following categories:

core            implementation classes, lots of strongly connected packages
api             api classes, do not refer to core, very few imports
bridge          refer strongly to core and have expensive imports


At this stage, the trick is to do some work by hand until you find you really 
understand the problem.

It would be perfect if you could take a look at Tinkerpop and JUNG. I think it 
would be quite easy to visualize the graph of dependencies.

You mean my next step is developing a Tinkerpop or JUNG graphical view
for this report for user to adjust the bundles details manully ? Am i
right ? I will start learn Jung and start this job soon

Kind regards,

        Peter Kriens




On 26 jul 2011, at 16:24, Tiger Gui wrote:

Hi Peter,

This is the whole application split algorithm here. After application
source code analyse algorithm described here[1], we can know each
package/class use which packages/classes and be used by which
packages/classes. Now, we just discuss package here, we treat package
as a single atom, each package has three important attributes, usedBy,
usesExternal and usesInternal, just like below:

<package name="org.apache.catalina.deploy"
sources="/E:/GSoC/gsoc2011osgi/runtime-New_configuration/TomcatJava/bin"
size="30" usedBy="8" usesInternal="11" usesExternal="14" layer="6"
cycle="org.apache.catalina et al.">
      <packageRef name="org.apache.catalina.core" type="usedBy"/>
      <packageRef name="org.apache.catalina.startup" type="usedBy"/>
      <packageRef name="org.apache.catalina.deploy" type="usedBy"/>
      <packageRef name="org.apache.catalina.authenticator" type="usedBy"/>
      <packageRef name="org.apache.catalina" type="usedBy"/>
      <packageRef name="org.apache.catalina.mbeans" type="usedBy"/>
      <packageRef name="org.apache.catalina.connector" type="usedBy"/>
      <packageRef name="org.apache.catalina.realm" type="usedBy"/>
      <packageRef name="java.lang" type="usesExternal"/>
      <packageRef name="java.io" type="usesExternal"/>
      <packageRef name="org.apache.catalina.deploy" type="usesInternal"/>
      <packageRef name="org.apache.catalina.util" type="usesInternal"/>
      <packageRef name="java.util" type="usesExternal"/>
      <packageRef name="org.apache.juli.logging" type="usesInternal"/>
      <packageRef name="org.apache.tomcat.util.res" type="usesInternal"/>
      <packageRef name="java.beans" type="usesExternal"/>
      <packageRef name="(default package)" type="usesExternal"/>
      <packageRef name="org.apache.catalina" type="usesInternal"/>
      <packageRef name="org.apache.catalina.mbeans" type="usesInternal"/>
      <packageRef name="javax.management" type="usesExternal"/>
      <packageRef name="namingResources" type="usesExternal"/>
      <packageRef name="javax.naming" type="usesExternal"/>
      <packageRef name="org.apache.naming" type="usesInternal"/>
      <packageRef name="java.lang.reflect" type="usesExternal"/>
      <packageRef name="javax.servlet" type="usesInternal"/>
      <packageRef name="javax.servlet.annotation" type="usesInternal"/>
      <packageRef name="org.apache.catalina.order" type="usesExternal"/>
      <packageRef name="java.net" type="usesExternal"/>
      <packageRef name="webXml" type="usesExternal"/>
      <packageRef name="webXml.version" type="usesExternal"/>
      <packageRef name="webxml" type="usesExternal"/>
      <packageRef name="org.apache.catalina.core" type="usesInternal"/>
      <packageRef name="javax.servlet.descriptor" type="usesInternal"/>
    </package>

usedBy means who use current package;
usesInternal means which packages current package use in current
application source code;
usesExternal means which packages current pakcage use not in current
application's source code.

Now, we start split the whole application in to bundles, according to
above algorithm source code analyse algorithm, we can also know
package cycles in current application.

1. Treat each package cycle as a single bundle;
2. Treat each packages not in cycle as a single bundle;
3. Then we should decide which bundles can be merged into one new bundle;

First round merge job:
4. If one bundle's all usesInternal elements are in the other bundle,
these two bundles should be merged into a new bundle;

Think about how to merge used by only bundle (bundle be used only by
other bundle, it does not rely on any other bundle):
Define two variable for used by only bundle:
sameUB: it means the number of two bundles have same usedBy elements.
sameUE: it menas the number of two bundles have same usedExternal
external package rely elements.

boolean condition1 = 4 * sameUB>= one.usedByList.size() +
two.usedByList.size();
boolean condition2 = 2 * sameUE>= one.usedExternalList.size() +
two.usedExternalList.size();

if condition1 or condition2 is true, we should merge these two usedBy
only bundle.

Merge the other bundles:
5.The core problem is how to decide two bundles(bundle one and bundle
two) can be merged or not.

Define 5 variables:
uiNumber: the sum of bundle one's usesInternal elements in bundle two
number and bundle two's usesInternal elements in bundle one number;

ubNumber:the sum of bundle one's usedBy elements in bundle two number
and bundle two's usedBy elements in bundle one number;

sameUI: the same usesInternal number bundle one and two have

sameUB: Be similar with used by only bundle's this variable

sameUE: Be similar with used by only bundle's this variable

Define these conditions:
boolean condition1 = 2 * uiNumber>= one.usesInternalList.size() +
two.usesInternalList.size();
boolean condition2 = 2 * ubNumber>= one.usedByList.size() +
two.usedByList.size() ;
boolean condition3 = 3.5 * sameUI>= one.usesInternalList.size() +
two.usesInternalList.size();
boolean condition4 = 4 * sameUB>= one.usedByList.size() +
two.usedByList.size();
boolean condition5 = 3 * sameUE>= one.usedExternalList.size() +
two.usedExternalList.size();

If any above condition is true, these two bundles can be merged. But
these are another problem, if bundle A can be merged with B, but it
also can be merged with C, now, we should decide merge A with B or A
with C.

Define the follow attribute:
int mergeFactor= 2 * (uiNumber + ubNumber) + 0.4 * (sameUI + sameUB) +
0.2 * sameUE - number;

If A and B's mergeFactor is 20 and A and C's mergeFactor is 30, we
should merge A and C.

This is current merge algorithm in OSGiMaker, i will keep on improving
it, use class relationship factors or etc. You can find the source
code detail of this algorithm in class AnalyseJob of our project.

The attach file is the analyse result document OSGiMaker analyse
Tomcat's source code and split it into bundles, you can have a review.


In fact, i did not want to bother you too much, but it seems that you
have enough time to help me to improve it, this is a good thing. If
you have any advises, please let me know, let's improving it together
:-)

Thank you

[1] 
http://code.google.com/p/osgimaker/wiki/Implement_of_project_analyse_algorithm

2011/7/26 Peter Kriens<peter.kri...@aqute.biz>:

Well, I do not think I am eager but I have a hard time getting a feeling where 
you are. You do not have to send reports, I want to see intermediate results 
and discuss issues if there are any. As I said earlier, it is not clear to me 
yet what the best algorithm is so that need to be worked out before we do the 
gui stuff.

Kind regards,

        Peter Kriens



On 26 jul 2011, at 11:24, Tiger Gui wrote:

In my schedule, i will report current status to you tomorrow as i
think i can get a usable version today,  the whole analyse and split
algorithm is complex i have to organize a document to describe it
clearly. As you are really eager to see its progress, it is OK, i will
start the report now

2011/7/26 Peter Kriens<peter.kri...@aqute.biz>:

If that is the case you have to provide more on going feedback. What happened 
to the analysis by hand?

Kind regards,

        Peter Kriens


On 26 jul 2011, at 11:02, Tiger Gui wrote:

No, Peter, i am really working hard for this project, you can check
the progress here[1]. Now, this tool can analyse a whole project and
export the its analyse result to bundles( I test it in Spring and
Tomcat project), if possible, you can install it in your Eclipse and
have a trial of it. But, i am still improving the split algorithm as
the current algorithm is not working perfect.

I will supply a document about the current status of this project and
a simple guide for you to have a trial of it. I am really working very
hard for it these days :-(

[1] http://code.google.com/p/osgimaker/updates/list

2011/7/26 Peter Kriens<peter.kri...@aqute.biz>:

I am getting the feeling that you're not working very hard on this project and 
only does something just for the evaluations ...

        Peter Kriens

--
Best Regards
----------------------------------------------------
Tiger Gui [tigergui1...@gmail.com]

--
Best Regards
----------------------------------------------------
Tiger Gui [tigergui1...@gmail.com]

--
Best Regards
----------------------------------------------------
Tiger Gui [tigergui1...@gmail.com]
<TomcatJava.analyse>

--
Best Regards
----------------------------------------------------
Tiger Gui [tigergui1...@gmail.com]
<bundles_relation.txt><SpringSplitTest.analyse>

--
Best Regards
----------------------------------------------------
Tiger Gui [tigergui1...@gmail.com]
<SpringSplitTest.html>

--
Best Regards
----------------------------------------------------
Tiger Gui [tigergui1...@gmail.com]

--
Best Regards
----------------------------------------------------
Tiger Gui [tigergui1...@gmail.com]
<TomcatJava.html>

--
Best Regards
----------------------------------------------------
Tiger Gui [tigergui1...@gmail.com]

--
Best Regards
----------------------------------------------------
Tiger Gui [tigergui1...@gmail.com]
<TomcatJava.html><tomat.png><tomcat2.png>

--
Best Regards
----------------------------------------------------
Tiger Gui [tigergui1...@gmail.com]

--
Best Regards
----------------------------------------------------
Tiger Gui [tigergui1...@gmail.com]