On 12/04/2012, at 8:40 AM, Adam Murdoch wrote:
>
> On 11/04/2012, at 6:48 PM, Luke Daley wrote:
>
>>
>> On 11/04/2012, at 12:23 AM, Adam Murdoch wrote:
>>
>>> On 10/04/2012, at 7:49 PM, Luke Daley wrote:
>>>
>>>> http://forums.gradle.org/gradle/topics/signing_plugin_add_to_configuration_even_if_its_not_being_run
>>>>
>>>> The signing plugin adds PublishArtifacts for each signature file. These
>>>> signature files may not actually be created depending on the signing
>>>> configuration:
>>>>
>>>> signing {
>>>> required { project.taskGraph.hasTask("uploadArchives") }
>>>> sign configurations.archives
>>>> }
>>>>
>>>> This “required” handling makes it easy to have a build that works for
>>>> people who don't have their environment set up for signing. If signing is
>>>> not required, and we can't sign, the signing task(s) is skipped so we end
>>>> up with the PublishArtifacts on the configuration pointing to non existent
>>>> files.
>>>>
>>>> The problem comes in that the Artifactory plugin tries to upload these non
>>>> existent files and explodes.
>>>>
>>>> One idea is to allow this as part of the model. That is, a PublishArtifact
>>>> becomes something that might be created during the build instead of
>>>> something that will be created.
>>>
>>> I think the problem here is that the signatures aren't really artefacts, in
>>> the sense that the point of the build is not to generate signatures, the
>>> point of the build is to build the jar or war or whatever it is that is
>>> being signed. Signatures are more like meta-data about artefacts, that are
>>> used as part of shipping the real artefacts around between machines and
>>> teams and so on.
>>>
>>> I would instead think about splitting publications up into 2 pieces:
>>> * The definition of the component that is being produced. This defines the
>>> artefacts that are fundamental to the component that always need to be
>>> build: the jar or the dll or the distribution zip or whatever.
>>> * The definition of the destination-specific publication. This defines the
>>> artefacts that will be published to the given destination, where
>>> destination is a repository. This would (by default) include the artefacts
>>> of the components being published. It might also include signatures or
>>> checksums or meta-data artefacts or whatever additional artefacts need to
>>> be published to this destination.
>>>
>>> This way, there's no need for conditional signing or PublishArtifacts. If
>>> you are building locally, the signing is not triggered. If you've declared
>>> that signatures are required to publish to the corporate repository, then
>>> you need to have signing set up if you want to publish there and its a
>>> failure if you don't.
>>
>> I'm not sure about expressing the requiredness of something purely as a
>> function of where it's going. That's too rigid in my opinion. One example is
>> that you may want to say that signatures are only required if we are doing a
>> final release, and you may not be using separate destinations for final and
>> non-final releases. I think it's much safer to allow the user to control
>> this for whatever case they need. We can apply patterns over the top of
>> course.
>>
>> Also, rather than destination-specific publications I think we should be
>> aiming for aggregate & filtered publications which can then be used in
>> whichever way the user wants. Tying it to a destination seems like
>> inflexible coupling to me.
>
> We're talking about the same thing here, I think. I've described the pattern,
> with the implicit assumption that it sits on top of an un-opionated toolkit.
> You've described the toolkit, with (I'm guessing) the assumption that an
> opinionated convention sits on top of it.
Indeed.
>> I agree that we need to be able to model metadata, but I think that's an
>> entirely separate issue. It's feasible to me that the build might not
>> produce first class artifacts under certain conditions and that being ok.
>> You can do this right now of course by never modelling the optional thing if
>> we aren't going to make it, but that implies that you can make the decision
>> reasonably early which may not be the case.
>
> I'm pretty sure that the optional things are a function of what you're
> building, and not a function of who you are or where you are.
I'm not sure that's always the case. Think of very tightly controlled
environments where who builds something is of utmost concern. I can conceive of
a case where you want the build to adapt to this so that you don't need to
model publication for every permutation.
As for where, I think there's a case for that too but it's more about
convenience. I want to run `gradle build` and have it build everything it can.
It may not be able to build some artifacts based on the environment.
I'm not saying that we need first class support for these concepts at the
opinionated level, but some way to achieve it would be nice. The location issue
is probably moot because you can determine up front and build the model
adaptively due to the static nature and inexpense of figuring this stuff out.
It may not be the same for the who. Doing permission checks may be expensive
and you may need to defer to execution time.
I still think there is a case for modelling something that might be created
depending on context.
> I also think there's generally only a small number of ways a given project
> needs to be published. So, for example, you might have a local build, a CI
> build or a release build, or in other words, 3 different combinations of
> artefacts.
For the simple case yes, but this can easily balloon when you are producing
multiple configurations of the software or dealing with many different test/qa
environments. It may be an anti-pattern, but it's a reality. This doesn't
really detract from anything you say below though.
> The solution, to me, is to use composition to model this. First, you model
> the things that the project inherently produces: a java library, a native
> application, whatever. These are the 'components' that the project produces.
> For each such component, you would define the artifacts that represent the
> component, that you need in order to use the component: the jar, the
> executable or install image or whatever. This would not include auxiliary
> artifacts like signatures or checksums. Implementation-wise, a component
> would be modelled as something like a buildable set of publish artefacts +
> some dependency meta-data. So, we have something like this:
>
> PublishArtifactsSet is-a Buildable
> PublishArtifactSet has-a set of PublishArtifacts
> ConfigurablePublishArtifactSet is-a PublishArtifactSet that allows
> composition, etc, similar to ConfigurableFileCollection.
> Component has-a ConfigurablePublishArtifactSet
> Project has-a set of named Components
>
> Then, you model the different ways you want to publish these components: you
> might have, say, a local and a release publication. Each publication is
> essentially just a set of artefacts. Implementation-wise, a publication is
> modelled as a PublishArtifactSet, and is usually a transformation of some
> other PublishArtifactSet (typically a Component, but could be anything that
> can be converted into a set of artefacts), to add in signatures and checksums
> and whatever else. So we have something like this:
>
> Publication has-a ConfigurablePublishArtifactSet
> SigningArtifactSet is-a PublishArtifactSet
> SigningArtifactSet transforms a PublishArtifactSet, and contains a signature
> artifact for each artefact in the original set.
> ChecksumArtifactSet, PomArtifactSet, IvyXmlArtifactSet do similar things.
> Project has-a set of named Publications
>
> Finally, you wire together some tasks to publish these publications to
> various destinations. Implementation-wise, a publish task would take an
> ArtifactRepository and a PublishArtifactSet, and would push the given
> artefacts to the given destination. Probably there would be a bunch of task
> rules, as we have now, perhaps something like:
>
> assemble${componentName} - builds the artefacts of the component
> (alternatively, the rule might just be ${componentName})
> assemble${publicationName} - builds the artefacts of the publication
> publish${publicationName} - builds and publishes the artefacts of the
> publication to all repositories
> assemble - builds all components
> publish - builds and publishes all publications.
>
> So, we have all the pieces. Out of the box, there'd be no components or
> publications defined.
>
> Applying a convention plugin (java, cpp-exe, application, war, etc) would add
> components of the appropriate type. The java plugin would add a java library,
> war plugin would add a web application, and so on.
>
> Applying the maven plugin would add a maven-style publication for each
> component, and probably define publish tasks that attach each maven
> publication to each maven repository.
>
> Applying the ivy plugin would add an ivy-style publication for each
> component, and the appropriate publish tasks.
>
> Applying the signing base plugin (whether that's the current signing plugin
> or a new one) would add the capability to sign Publications (or probably
> PublishArtifactSet more generally), but not actually add any signatures.
> Applying the signing convention plugin , would add signatures to each remote
> publication.
>
> This way, you can go with the conventions, or compose your own model. There's
> no need for conditional signing or artefacts, as everything you need is just
> a dependency of what you're building.
This all sounds good, but quite a way from where we are today. How & when do we
get there? And what do we do in the meantime to solve the immediate problem?
We should at least put some constraints on PublishArtifact one way or the
other. That is, (1) define it as an illegal state if an artifact file does not
exist after it's dependencies have been satisfied or (2) define this as legal
and specify what it means.
If we go with (1), we need an interim solution so the signing plugin can be
redesigned to accommodate.
Another option is that we ask the artifactory plugin guys to code in a special
check for signature artifacts where the signature file doesn't exist as an
interim measure.
> Your 'releaseBuild' task can depend on 'publishRelease', your 'ciBuild' task
> can depend on 'publishSnapshot', and your 'devBuild' task can depend on
> 'assemble'.
I don't think this changes your point, but modelling kinds of builds as tasks
just plain doesn't work (as you yourself have said). With build types, this
would probably just change to having the build type execute on of these tasks
instead of “depending” on them.
--
Luke Daley
Principal Engineer, Gradleware
http://gradleware.com
