On Sep 25, 2009, at 2:10 AM, Adam Murdoch wrote:
Hi,
It sounds to me like the generic solution might actually be easier
than the hard-coded solution, once you chase down all the edge
cases, and will also end up more accurate and reusable. Given that
we want to throw away the hard-coded solution as soon as 0.8 is out
and replace it with a generic solution, I wonder if it's worth
pursuing the hard-coded solution at all.
Hans Dockter wrote:
Hi,
I have implemented a task optimization functionality that we might
put into 0.8. I have uploaded my branch to: http://github.com/hansd/gradle/tree/optim
A couple of comments:
1.) The task history is now stored in gradle user home with some
hash that relates it to the actual project. The base for the hash
is the path of the root dir. We might have issues if a subproject
takes part in multiple multi-project builds, if the output is
sensitive to the respective multi-project build. The only way I see
to solve such a problem, would be to have multiple output dirs.
We want a unique identifier for the build, not for the project. At
this stage, the settings dir path would do. Or the project dir of
the root project.
That's the way it is done (I was not precise enough, when I said
'actual project' above. It is the build.). The base for the hash is
the path of the root dir.
2.) Each task has a now doesOutputExists() method which defaults to
false. So far all archive tasks have a custom implementation which
checks for the existence of the archive. The test task also has a
custom implementation which checks for at least one test results
file. I hope that we find a way to automate this in 0.9 by
introducing a generic notion of task output.
We already have the notion to some degree: properties can be marked
with @OutputFile and @OutputDirectory. The default doesOutputExists
() could make use of these.
Right.
3.) So far there are onlyIf implementations only for the test and
the jar task provided by the Java plugin. I will add an onlyIf
modification for the test task when the Groovy plugin is applied
tomorrow. For 0.9 we want to automate the onlyIf statements based
on the information we have on the input arguments of a task.
4.) What about the other tasks? For java compile the Ant javac task
has its own optimization checking for changed files. I'm not sure
about groovyc, I need to check. The Ant Javadoc/Groovydoc tasks do
not check for changed files. To optimize them we would need to
check for changed source files. The same is true for the code
quality stuff. I'm not sure whether I will have time to get this
done before 0.8. I would use Tom's change detection stuff. I
haven't had a look at that yet. For 0.9 I guess the SourceSet's
will be a good place for source change detection. For 0.8 it might
be already good enough to distinguish between no changes/do nothing
and do the full thing.
I think you can pretty quickly do something general for all tasks
with file inputs:
- In the onlyIf predicate, calculate the set of (file path,
timestamp) for all input files in the history. You could create a
hash from this.
- In the onlyIf predicate, skip the task if the input files hash ==
the input files hash from last successful execution and
task.doesOutputExists()
- execute the task
- store the input files hash in the history.
Yes.
5.) The onlyIf optimization needs to be disabled if any
build.gradle which is part of the multi-project build, the
settings.grade or an init.gradle changes. Therefore a ScriptSource
object now has a method hasChanged which defaults to true. The
DefaultScriptCompilerFactory sets it to false if a script is read
from the cache. I'm not very happy about the latter mechanism. To
me this looks like a hint that the ScriptSource should be
responsible for the compilation, instead of the compile class
having a side effect on the state of ScriptSource. I will think
about this in more detail tomorrow.
I think a better approach is to use the properties of the task. This
is more accurate, in that it catches changes to the task
configuration that aren't the result of changes to the build/init/
settings scripts. Some types of changes we don't catch by checking
if the scripts has changed:
* Task is configured using -PsomeProperty=value, and that value is
different to last execution.
* Task is configured using system property, and that value is
different to last execution.
* Task is configured based on the DAG, and the DAG contains
different tasks to last execution.
* Task is configured by a 3rd party plugin, and that plugin has
changed since last execution
* Task is configured by buildSrc code, and that code has changed
since last execution
* Task is configured using properties from an imported build.xml,
and that build.xml has changed
* Task is configured using properties from gradle.properties, ...
* ... you get the idea ...
So, checking whether the scripts have changed since last execution
doesn't come close to accurately detecting if we need to re-execute
a task. It also means we unnecessarily re-execute tasks when an
unrelated change has been made to the build script.
I think accuracy is really important with this stuff. It absolutely
must be reliable, or people will just run clean all the time to get
a reliable build. We want to avoid this.
I would suggest instead that we add an @Input annotation which one
can use to mark up the properties of a task which contribute in some
significant way to the output of the task. The input of a task is
stored in the history, and the set of input files is simply treated
as one piece of input.
I agree. I guess that is what we should do. And with the annotations
it looks rather straight forward to implement.
6.) The GradleInternal class exposes now the settings and the init
script ScriptSource objects. It also provides a convenience method
to check whether any ScriptSource object has changed. To get hold
of the settings object it registers as a BuildListener. I think
there should be a better way. I will think more about this tomorrow.
Remove the settings file, perhaps? :)
I'm not completely sure whether we want to push this into 0.8 or
not. Feedback is welcome.
I don't think it will be reliable enough.
I also think we should leave it out for 0.8.
- Hans
--
Hans Dockter
Gradle Project Manager
http://www.gradle.org
---------------------------------------------------------------------
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
