On 20/08/2012, at 9:43 PM, Luke Daley wrote: > > On 20/08/2012, at 5:36 AM, Adam Murdoch wrote: > >> >> On 18/08/2012, at 1:15 AM, Luke Daley wrote: >> >>> Howdy, >>> >>> There's an aspect of the migration comparison/verification work that could >>> benefit from some focused discussion. It's mostly an implementation level >>> issue so this conversation can run in parallel to other work in this space. >>> >>> At a basic level what we need to do is run two builds and compare _things_ >>> about this. What I want to try and lock down is exactly what these “things” >>> can be. There's two prevailing schools of thought; these things are always >>> files, or they are much more abstract and could be anything. A component of >>> this work will be delivering an API that allows the user to assemble their >>> own comparison process (for non trivial cases). If everything is a file the >>> API could be simpler, *maybe* in a significant way. >>> >>> In the migration from tool X (Ant or Maven) to Gradle, tool X is more or >>> less an opaque box that we poke and then it does something. Our options for >>> understanding what it did are its process exit value, its stdio output and >>> what it leaves on the filesystem. In a Gradle to Gradle migration (i.e. >>> upgrade), we have much richer information exchange potential. >>> >>> If we decide that _all_ we want to do with this feature is compare the >>> files that builds generate then the issue is moot. We can obviously then >>> design the entire API around comparing files. >>> >>> I'm not sure we want to do make this restriction though. I think there's a >>> case for comparing other things. >>> >>> Consider something like a classpath. In the case of a migration from a >>> different system, it might be useful to understand how the compile >>> classpath is different for an artifact during the development of the Gradle >>> build being migrated to. Unless we did this automatically (which we >>> probably could do for Maven but not Ant) I doubt many users would bother >>> with this. It seems more compelling in the Gradle upgrade case though. >>> That's definitely something I'm going to want to be aware of changing if it >>> does. >>> >>> There are some other Gradle specific things we could compare: >>> >>> * All of the tasks in a project (i.e. be told that a Gradle upgrade >>> introduces or removes tasks that were in the build before) >>> * All of the DSL extensions in a project (this one is pretty dubious I >>> think) >>> * The set of configurations/archives etc. >>> >>> Upgrade verification isn't just about giving a binary yes/no answer. Part >>> of it will be understanding differences introduced by new versions, which >>> means deeper analysis. When (if?) we start using this feature to help >>> people test speculative changes to their build, this becomes even more >>> important. >>> >>> We also may use this “comparison” toolkit in our own QA, which is another >>> argument for being more general/abstract. >>> >>> >>> My current thinking is that at the base of the API we need to stay general >>> and be able to compare just about anything (by plugging in different >>> strategies), and then layer a file oriented set of strategies on top. >> >> I think anything beyond comparing files and text files is unnecessary, and >> overcooking it a bit. Here's why: >> >> What we're doing here is adding a smoke test that verifies that a change to >> the build system is just as likely to not break things, as any other change >> (changing the code, tweaking a configuration file, upgrading a dependency, >> etc). We don't have to do anything more than say 'according to your current >> quality checks, this build works as well as that build'. The goal isn't to >> solve the world here, or implement some general purpose comparison engine. >> >> Instead, we need to do 2 things: >> >> 1. Encourage people to write tests for the build outcomes. Then, when we >> want to verify a change from one build to another, we run the test for both >> builds. If it passes in both builds, then the outcome is the same. >> >> This way, you continue to verify that your build works beyond the lifetime >> of any migration you might undertake. If it's important enough to compare >> during migration, it's important enough to verify every time something else >> changes. > > Can you give some examples of the kinds of tests you are thinking of here > please. I'm not sure I'm on your wavelength.
* Is the web app deployed and usable, on the expected machines? * Is the database set up with the correct schema and test data? (could probably just be inferred from the above) * Are the published artefacts resolvable and usable in another build? * Has the source and documentation been published? * Is the correct tag on the correct VCS revision? * Does the documentation have the correct version in it? * Is the license and documentation included in the distribution zip? * Is the sample included in the distribution zip usable? * Does the manifest of each jar have the project name and version in it? * Whatever else you've broken in the past: not including debugging information, generating java 6 byte code instead of java 5 byte code, linking a DLL against the single threaded runtime instead of the multi threaded runtime, forgetting to include a css file in the documentation zip, and so on. > >> 2. Be able to compare certain key types of files. In particular, various >> archives and text files. >> >> This way, if you've something about the build you need to verify, you >> generate a text file from each build and we can do a text diff on the >> contents and tell you if they are the same. > > I think this the key thing. If people can easily enough represent important > internal structures as text files (and I can't see why not), then this > strategy should be fine. > >> So, we have 2 extensible mechanisms here: making use of tests, and comparing >> text files. I think in practise this will be plenty. >> >> Of course, the implementation will have some kind of pluggable strategies, >> or abstractions, or what ever. This is fine, but we should keep it internal. >> We shouldn't expose this until we know we need it. It is essential that we >> keep the public API as focused, and concrete, as possible. > > For system X to Gradle migration I think this is right, which means that our > public API toolkit can focus on a file based API. Let's consider this issue > resolved. > > As I think you are saying, it makes sense to layer this on top of a more > general abstraction though. For the Gradle to Gradle case I would hope over > time that our comparison inputs become much richer to give people greater > confidence about upgrades and for this _having_ to go to a file would be > unnecessary indirection. Absolutely. And we may run into use cases where it makes sense to make this a public capability. Just not yet. > > So it might be necessary to effectively have two APIs here and adapt the > public API to the internal, more general, API (as opposed to the public > component being a subset). I think this is a good general approach, and something we do in a bunch of places. -- Adam Murdoch Gradle Co-founder http://www.gradle.org VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com
