On 18/08/2012, at 1:15 AM, Luke Daley wrote: > Howdy, > > There's an aspect of the migration comparison/verification work that could > benefit from some focused discussion. It's mostly an implementation level > issue so this conversation can run in parallel to other work in this space. > > At a basic level what we need to do is run two builds and compare _things_ > about this. What I want to try and lock down is exactly what these “things” > can be. There's two prevailing schools of thought; these things are always > files, or they are much more abstract and could be anything. A component of > this work will be delivering an API that allows the user to assemble their > own comparison process (for non trivial cases). If everything is a file the > API could be simpler, *maybe* in a significant way. > > In the migration from tool X (Ant or Maven) to Gradle, tool X is more or less > an opaque box that we poke and then it does something. Our options for > understanding what it did are its process exit value, its stdio output and > what it leaves on the filesystem. In a Gradle to Gradle migration (i.e. > upgrade), we have much richer information exchange potential. > > If we decide that _all_ we want to do with this feature is compare the files > that builds generate then the issue is moot. We can obviously then design the > entire API around comparing files. > > I'm not sure we want to do make this restriction though. I think there's a > case for comparing other things. > > Consider something like a classpath. In the case of a migration from a > different system, it might be useful to understand how the compile classpath > is different for an artifact during the development of the Gradle build being > migrated to. Unless we did this automatically (which we probably could do for > Maven but not Ant) I doubt many users would bother with this. It seems more > compelling in the Gradle upgrade case though. That's definitely something I'm > going to want to be aware of changing if it does. > > There are some other Gradle specific things we could compare: > > * All of the tasks in a project (i.e. be told that a Gradle upgrade > introduces or removes tasks that were in the build before) > * All of the DSL extensions in a project (this one is pretty dubious I think) > * The set of configurations/archives etc. > > Upgrade verification isn't just about giving a binary yes/no answer. Part of > it will be understanding differences introduced by new versions, which means > deeper analysis. When (if?) we start using this feature to help people test > speculative changes to their build, this becomes even more important. > > We also may use this “comparison” toolkit in our own QA, which is another > argument for being more general/abstract. > > > My current thinking is that at the base of the API we need to stay general > and be able to compare just about anything (by plugging in different > strategies), and then layer a file oriented set of strategies on top.
I think anything beyond comparing files and text files is unnecessary, and overcooking it a bit. Here's why: What we're doing here is adding a smoke test that verifies that a change to the build system is just as likely to not break things, as any other change (changing the code, tweaking a configuration file, upgrading a dependency, etc). We don't have to do anything more than say 'according to your current quality checks, this build works as well as that build'. The goal isn't to solve the world here, or implement some general purpose comparison engine. Instead, we need to do 2 things: 1. Encourage people to write tests for the build outcomes. Then, when we want to verify a change from one build to another, we run the test for both builds. If it passes in both builds, then the outcome is the same. This way, you continue to verify that your build works beyond the lifetime of any migration you might undertake. If it's important enough to compare during migration, it's important enough to verify every time something else changes. 2. Be able to compare certain key types of files. In particular, various archives and text files. This way, if you've something about the build you need to verify, you generate a text file from each build and we can do a text diff on the contents and tell you if they are the same. So, we have 2 extensible mechanisms here: making use of tests, and comparing text files. I think in practise this will be plenty. Of course, the implementation will have some kind of pluggable strategies, or abstractions, or what ever. This is fine, but we should keep it internal. We shouldn't expose this until we know we need it. It is essential that we keep the public API as focused, and concrete, as possible. -- Adam Murdoch Gradle Co-founder http://www.gradle.org VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com
