[gwt-contrib] Re: Some random modularization/cleanup-related news
On Fri, Nov 15, 2013 at 11:56 AM, Matthew Dempsky wrote:
> If I first collapse all source files into a single vertex, then our 502
> packages still form 394 SCCs.
>
Oops, that should be "collapse all source files *within a Java package*
into a single vertex". (Collapsing all source files should of course yield
1 SCC. ;))
Also, for anyone interested in replicating my results on their own,
attached is my simple Tool.java program. Invoke it like:
TOOLS=$(dirname $(which javac))/../lib/tools.jar
DEPS=...colon separated list of GWT_TOOLS jars to compile against...
SRCS=$(find user/src user/test dev/core/src dev/codeserver/java
dev/core/test dev/core/super '(' -name super -o -name intrinsic ')' -prune
-o -name '*.java' -print)
javac -cp ${TOOLS} Tool.java
java -cp ${TOOLS} Tool -cp ${DEPS} ${SRCS}
(I haven't actually tried this; I'm using Google's internal BUILD system
for simplicity, but the above should reasonably approximate what I'm doing.)
The output is a list of "file1: file2" lines, where file1 makes some
mention of a symbol defined in file2. I've then been post processing with
Python to do SCC analysis.
--
http://groups.google.com/group/Google-Web-Toolkit-Contributors
---
You received this message because you are subscribed to the Google Groups "GWT
Contributors" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-web-toolkit-contributors+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
import com.sun.source.tree.CompilationUnitTree;
import com.sun.source.tree.Tree;
import com.sun.source.util.JavacTask;
import com.sun.source.util.TreeScanner;
import com.sun.tools.javac.code.Symbol;
import com.sun.tools.javac.code.Symbol.ClassSymbol;
import com.sun.tools.javac.tree.JCTree;
import com.sun.tools.javac.tree.TreeInfo;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import javax.tools.JavaCompiler;
import javax.tools.JavaCompiler.CompilationTask;
import javax.tools.StandardJavaFileManager;
import javax.tools.ToolProvider;
public class Tool {
public static void main(String[] args) throws Exception {
List argv = Arrays.asList(args);
JavaCompiler jc = ToolProvider.getSystemJavaCompiler();
StandardJavaFileManager fm = jc.getStandardFileManager(null, null, null);
JavacTask task = (JavacTask) jc.getTask(null, fm, null, argv.subList(0, 2), null,
fm.getJavaFileObjectsFromStrings(argv.subList(2, argv.size(;
Iterable trees = task.parse();
task.analyze();
for (CompilationUnitTree tree : trees) {
final Set refs = new HashSet();
tree.accept(new TreeScanner() {
@Override
public Void scan(Tree node, Void v) {
// Scan recurses until it hits null.
// Not all tree nodes have associated symbols.
// Package symbols don't have an enclosing class.
// Builtin symbols (e.g., Array) have no source or class file.
if (node != null) {
Symbol sym = TreeInfo.symbol((JCTree) node);
if (sym != null) {
ClassSymbol enclSym = sym.enclClass();
if (enclSym != null) {
if (enclSym.classfile != null) {
refs.add(enclSym.classfile.getName());
} else if (enclSym.sourcefile != null) {
refs.add(enclSym.sourcefile.getName());
}
}
}
}
return super.scan(node, v);
}
}, null);
String fn = tree.getSourceFile().getName();
for (String ref : refs) {
System.out.println(fn + ": " + ref);
}
}
}
}
Re: [gwt-contrib] Some random modularization/cleanup-related news
sounds *very* promising On Fri, Nov 15, 2013 at 11:56 AM, Matthew Dempsky wrote: > Last night I wrote a quick one-off javac-based tool to parse and analyze > all of GWT's source code, and to extract out inter-file dependencies (e.g., > dev/core/src/com/google/gwt/dev/GWTShell.java depends on > dev/core/src/com/google/gwt/dev/cfg/ModuleDef.java). I then ran a > strongly-connected components analysis on the resulting graph to form a DAG. > > My expectation was our interdependencies would be hairy and cyclic enough > that the DAG would probably already be pretty constrained. But it turns > out that our 6022 source files form 5206 SCCs. If I first collapse all > source files into a single vertex, then our 502 packages still form 394 > SCCs. Ignoring test sources, I still get 246 packages grouped into 138 > SCCs. > > I think that's still too many to be immediately useful, so probably my > next step is to write a tool to let me interactively find components that > share a lot of incoming and outgoing edges and merge them together. > > If anyone has suggestions on additional criteria I can easily take into > account, I'm interested. (E.g., I know I should probably take into account > super source and/or gwt.xml files, but javac doesn't know about those, so I > just skipped them for now for simplicity. ;)) > > > Also, as something maybe interesting, here's a map from our third party > libraries to source directories where we use them. (For simplicity, I'm > running this against the copy of GWT we're using within Google; e.g., we > haven't merged the HtmlUnit/Jetty/JSON changes yet. Oh, and this skips > JUnit and Selenium for no particular reason, but JUnit uses are pretty > obvious and Selenium is only used by RunStyleSelenium.) > > lib/apache/ant-1.6.5.jar: > dev/core/src/com/google/gwt/dev/resource/impl > dev/core/src/com/google/gwt/dev/shell/jetty > dev/core/test/com/google/gwt/dev/resource/impl > > lib/apache/commons/commons-collections-3.2.1.jar: > dev/core/test/org/apache/commons/collections/iterators > dev/core/test/org/apache/commons/collections/map > user/test/com/google/gwt/i18n > user/test/org/apache/commons/collections > > lib/apache/tapestry-util-text-4.0.2.jar: > user/src/com/google/gwt/i18n/rebind > > lib/cldr/21/icu4j.jar: > user/src/com/google/gwt/i18n/rebind > > lib/easymock/easymock-3.0.jar: > user/test/com/google/gwt/resources/rg > user/test/com/google/gwt/uibinder/rebind > > lib/eclipse/jdtCompilerAdapter-3.8.3.v20130121-145325.jar: > dev/core/src/com/google/gwt/dev/shell/jetty > > lib/eclipse/org.eclipse.jdt.core_3.8.3.v20130121-145325.jar: > dev/core/src/com/google/gwt/dev/javac > dev/core/src/com/google/gwt/dev/jdt > dev/core/src/com/google/gwt/dev/jjs/ast > dev/core/src/com/google/gwt/dev/jjs/impl > dev/core/test/com/google/gwt/dev/javac > > lib/guava/guava-15.0/guava-15.0-rebased.jar: > dev/codeserver/java/com/google/gwt/dev/codeserver > dev/core/src/com/google/gwt/core/ext > dev/core/src/com/google/gwt/core/ext/linker/impl > dev/core/src/com/google/gwt/core/linker > dev/core/src/com/google/gwt/dev > dev/core/src/com/google/gwt/dev/cfg > dev/core/src/com/google/gwt/dev/javac > dev/core/src/com/google/gwt/dev/javac/asm > dev/core/src/com/google/gwt/dev/javac/typemodel > dev/core/src/com/google/gwt/dev/jjs > dev/core/src/com/google/gwt/dev/jjs/ast > dev/core/src/com/google/gwt/dev/jjs/ast/js > dev/core/src/com/google/gwt/dev/jjs/impl > dev/core/src/com/google/gwt/dev/jjs/impl/codesplitter > dev/core/src/com/google/gwt/dev/jjs/impl/gflow > dev/core/src/com/google/gwt/dev/jjs/impl/gflow/cfg > dev/core/src/com/google/gwt/dev/jjs/impl/gflow/constants > dev/core/src/com/google/gwt/dev/jjs/impl/gflow/copy > dev/core/src/com/google/gwt/dev/jjs/impl/gflow/liveness > dev/core/src/com/google/gwt/dev/jjs/impl/gflow/unreachable > dev/core/src/com/google/gwt/dev/js > dev/core/src/com/google/gwt/dev/js/ast > dev/core/src/com/google/gwt/dev/resource/impl > dev/core/src/com/google/gwt/dev/shell > dev/core/src/com/google/gwt/dev/util > dev/core/src/com/google/gwt/dev/util/arg > dev/core/src/com/google/gwt/soyc > dev/core/test/com/google/gwt/core/ext/linker/impl > dev/core/test/com/google/gwt/dev/cfg > dev/core/test/com/google/gwt/dev/javac > dev/core/test/com/google/gwt/dev/jjs/impl > dev/core/test/com/google/gwt/dev/jjs/impl/codesplitter > dev/core/test/com/google/gwt/dev/jjs/impl/gflow > dev/core/test/com/google/gwt/dev/jjs/impl/gflow/cfg > dev/core/test/com/google/gwt/dev/js > user/src/com/google/gwt/editor/rebind > user/src/com/google/gwt/i18n/rebind > user/src/com/google/gwt/junit > user/src/com/google/gwt/resources/rg > user/src/com/google/gwt/safecss/shared > user/src/com/google/gwt/safehtml/rebind > user/src/com/google/gwt/safehtml/shared > user/src/com/google/gwt/uibinder/rebind > user/src/com/google/gwt/useragent/rebind > user/src/com/google/gwt/user/rebind > user
Re: [gwt-contrib] Some random modularization/cleanup-related news
I do care a lot about this topic: I'm new to the source code of GWT; I did started digging into it less than one month ago with the intent of ease use of GWT in OSGi environment. For this reason, I did a (simple) run of JDepend on the GWT source code, merged all together in a single project, to study packages dependencies. I got this report http://goo.gl/U688XQ that also gives an idea of the complexity. Another proof of the coupling comes from the final OSGi Import-Package and and Export-Package setup I had to use (see at the end) to make gwt-servlet.jar work as an independent OSGi bundle serving gwt examples from the distribution. My overall feeling is that the initial partition into User & Dev is no more sufficient to the size of the project, and I also judge a mistake that need to be fixed the way Servlet jar is formed by extracting classes from the two previous. I think it would help designing the ideal modularity of GWT with a mind map or in another breakdown structure form, so that new developments could follow a good orientation. If the vision is clear, people could refactor code gradually while working on other features and patches. Probably we will need to be prepared to breaking some backward compatibility (for example, gwt-servlet that is expected to depend only on server side or shared code, won't work without the package com.google.gwt.user.client.rpc and I think com.google.gwt.user.client.rpc should be moved to com.google.gwt.user.shared.rpc) and this could be only acceptable for a very major release (GWT 3.0.0?). If this has to be done, it has to be motivated by a clear path. In the end, as an example, take a look at the Apache Wicket project, it probably have the same size of GWT and the level of modularity of it source code seems good (Wicket has however a other design issues and I prefer a lot GWT ;-) ) Please go on and deep on this topic, Cristiano PS. OSGi's Import-Package and Export-Package necessary for gwt-servlet.jar. Note the setup may not be optimal, I did followed a Test Driven approach stopping as soon as each example was working. Import-Package = com.google.web.bindery.event.shared com.google.web.bindery.requestfactory.shared.messages javax.annotation javax.annotation.processing javax.imageio javax.imageio.metadata javax.imageio.stream javax.inject javax.lang.model javax.lang.model.element javax.lang.model.type javax.lang.model.util javax.servlet javax.servlet.http javax.tools javax.validation javax.validation.bootstrap javax.validation.constraints javax.validation.groups javax.validation.metadata javax.validation.spi junit.framework org.json org.w3c.css.sac org.w3c.dom org.w3c.flute.parser sun.misc Export-Package = com.google.gwt.activity.shared com.google.gwt.codegen.server com.google.gwt.core.server com.google.gwt.core.shared com.google.gwt.dev.asm com.google.gwt.dev.asm.commons com.google.gwt.dev.asm.signature com.google.gwt.dev.asm.tree com.google.gwt.dev.asm.util com.google.gwt.dev.protobuf com.google.gwt.dev.util com.google.gwt.dom.builder.shared com.google.gwt.event.logical.shared com.google.gwt.event.shared com.google.gwt.event.shared.testing com.google.gwt.i18n.linker com.google.gwt.i18n.server com.google.gwt.i18n.server.keygen com.google.gwt.i18n.server.testing com.google.gwt.i18n.shared com.google.gwt.junit.linker com.google.gwt.logging.server com.google.gwt.logging.shared com.google.gwt.place.shared com.google.gwt.precompress.linker com.google.gwt.regexp.shared com.google.gwt.resources.css com.google.gwt.resources.css.ast com.google.gwt.resources.ext com.google.gwt.resources.rg com.google.gwt.rpc.linker com.google.gwt.rpc.server com.google.gwt.safecss.shared com.google.gwt.safehtml.shared com.google.gwt.text.shared com.google.gwt.text.shared.testing com.google.gwt.thirdparty.debugging.sourcemap com.google.gwt.thirdparty.debugging.sourcemap.proto com.google.gwt.thirdparty.guava.common.annotations com.google.gwt.thirdparty.guava.common.base com.google.gwt.thirdparty.guava.common.base.internal com.google.gwt.thirdparty.guava.common.cache com.google.gwt.thirdparty.guava.common.collect com.google.gwt.thirdparty.guava.common.escape com.google.gwt.thirdparty.guava.common.eventbus com.google.gwt.thirdparty.guava.common.hash com.google.gwt.thirdparty.guava.common.html com.google.gwt.thirdparty.guava.common.io com.google.gwt.thirdparty.guava.common.math com.google.gwt.thirdparty.guava.common.net com.google.gwt.thirdparty.guava.common.primitives com.google.gwt.thirdparty.guava.common.reflect com.google.gwt.thirdparty.guava.common.util.concurrent com.google.gwt.thirdparty.guava.common.xml com.google.gwt.thirdparty.streamhtmlparser com.google.gwt.thirdparty.streamhtmlparser.util com.google.gwt.typedarrays.server com.google.gwt.typedarrays.shared com.google.gwt.uibinder.attributeparsers com.google.gwt.uibinder.elementparsers com.google.gwt.user.client.rpc com.google.gwt.user.client.rpc.core.com.google.gwt.core.shared com.google.gwt.user.client.rpc.core.java.lang com.g
[gwt-contrib] Re: Some random modularization/cleanup-related news
Interesting work. For an interactive solution you may try IntelliJ's DSM analysis: http://www.jetbrains.com/idea/features/dependency_analysis.html After getting used to it, it gives pretty good information about dependencies. Maybe it also knows about JSNI dependencies. -- J. -- http://groups.google.com/group/Google-Web-Toolkit-Contributors --- You received this message because you are subscribed to the Google Groups "GWT Contributors" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-contributors+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[gwt-contrib] Some random modularization/cleanup-related news
Last night I wrote a quick one-off javac-based tool to parse and analyze all of GWT's source code, and to extract out inter-file dependencies (e.g., dev/core/src/com/google/gwt/dev/GWTShell.java depends on dev/core/src/com/google/gwt/dev/cfg/ModuleDef.java). I then ran a strongly-connected components analysis on the resulting graph to form a DAG. My expectation was our interdependencies would be hairy and cyclic enough that the DAG would probably already be pretty constrained. But it turns out that our 6022 source files form 5206 SCCs. If I first collapse all source files into a single vertex, then our 502 packages still form 394 SCCs. Ignoring test sources, I still get 246 packages grouped into 138 SCCs. I think that's still too many to be immediately useful, so probably my next step is to write a tool to let me interactively find components that share a lot of incoming and outgoing edges and merge them together. If anyone has suggestions on additional criteria I can easily take into account, I'm interested. (E.g., I know I should probably take into account super source and/or gwt.xml files, but javac doesn't know about those, so I just skipped them for now for simplicity. ;)) Also, as something maybe interesting, here's a map from our third party libraries to source directories where we use them. (For simplicity, I'm running this against the copy of GWT we're using within Google; e.g., we haven't merged the HtmlUnit/Jetty/JSON changes yet. Oh, and this skips JUnit and Selenium for no particular reason, but JUnit uses are pretty obvious and Selenium is only used by RunStyleSelenium.) lib/apache/ant-1.6.5.jar: dev/core/src/com/google/gwt/dev/resource/impl dev/core/src/com/google/gwt/dev/shell/jetty dev/core/test/com/google/gwt/dev/resource/impl lib/apache/commons/commons-collections-3.2.1.jar: dev/core/test/org/apache/commons/collections/iterators dev/core/test/org/apache/commons/collections/map user/test/com/google/gwt/i18n user/test/org/apache/commons/collections lib/apache/tapestry-util-text-4.0.2.jar: user/src/com/google/gwt/i18n/rebind lib/cldr/21/icu4j.jar: user/src/com/google/gwt/i18n/rebind lib/easymock/easymock-3.0.jar: user/test/com/google/gwt/resources/rg user/test/com/google/gwt/uibinder/rebind lib/eclipse/jdtCompilerAdapter-3.8.3.v20130121-145325.jar: dev/core/src/com/google/gwt/dev/shell/jetty lib/eclipse/org.eclipse.jdt.core_3.8.3.v20130121-145325.jar: dev/core/src/com/google/gwt/dev/javac dev/core/src/com/google/gwt/dev/jdt dev/core/src/com/google/gwt/dev/jjs/ast dev/core/src/com/google/gwt/dev/jjs/impl dev/core/test/com/google/gwt/dev/javac lib/guava/guava-15.0/guava-15.0-rebased.jar: dev/codeserver/java/com/google/gwt/dev/codeserver dev/core/src/com/google/gwt/core/ext dev/core/src/com/google/gwt/core/ext/linker/impl dev/core/src/com/google/gwt/core/linker dev/core/src/com/google/gwt/dev dev/core/src/com/google/gwt/dev/cfg dev/core/src/com/google/gwt/dev/javac dev/core/src/com/google/gwt/dev/javac/asm dev/core/src/com/google/gwt/dev/javac/typemodel dev/core/src/com/google/gwt/dev/jjs dev/core/src/com/google/gwt/dev/jjs/ast dev/core/src/com/google/gwt/dev/jjs/ast/js dev/core/src/com/google/gwt/dev/jjs/impl dev/core/src/com/google/gwt/dev/jjs/impl/codesplitter dev/core/src/com/google/gwt/dev/jjs/impl/gflow dev/core/src/com/google/gwt/dev/jjs/impl/gflow/cfg dev/core/src/com/google/gwt/dev/jjs/impl/gflow/constants dev/core/src/com/google/gwt/dev/jjs/impl/gflow/copy dev/core/src/com/google/gwt/dev/jjs/impl/gflow/liveness dev/core/src/com/google/gwt/dev/jjs/impl/gflow/unreachable dev/core/src/com/google/gwt/dev/js dev/core/src/com/google/gwt/dev/js/ast dev/core/src/com/google/gwt/dev/resource/impl dev/core/src/com/google/gwt/dev/shell dev/core/src/com/google/gwt/dev/util dev/core/src/com/google/gwt/dev/util/arg dev/core/src/com/google/gwt/soyc dev/core/test/com/google/gwt/core/ext/linker/impl dev/core/test/com/google/gwt/dev/cfg dev/core/test/com/google/gwt/dev/javac dev/core/test/com/google/gwt/dev/jjs/impl dev/core/test/com/google/gwt/dev/jjs/impl/codesplitter dev/core/test/com/google/gwt/dev/jjs/impl/gflow dev/core/test/com/google/gwt/dev/jjs/impl/gflow/cfg dev/core/test/com/google/gwt/dev/js user/src/com/google/gwt/editor/rebind user/src/com/google/gwt/i18n/rebind user/src/com/google/gwt/junit user/src/com/google/gwt/resources/rg user/src/com/google/gwt/safecss/shared user/src/com/google/gwt/safehtml/rebind user/src/com/google/gwt/safehtml/shared user/src/com/google/gwt/uibinder/rebind user/src/com/google/gwt/useragent/rebind user/src/com/google/gwt/user/rebind user/src/com/google/gwt/user/rebind/rpc user/src/com/google/gwt/user/rebind/ui user/src/com/google/gwt/validation/rebind user/test/com/google/gwt/validation/rebind user/test/org/hibernate/jsr303/tck/util lib/hibernate/validator/hibernate-validator-4.1.0.Final.jar: user/src/com/go
