I'm going to be picking up the publication changes, so I wanted to
revisit this based on where I remember Danek and I ending up as well as
recent conversations I've had with Shawn where he made a case to put all
of the processing in the client as opposed to the server.
The one change I want to call out for review is that I've changed how
manifests are compared.(Basically, the new step 7 differs from the
previous step 6.) Take the following example:
A@1 is published. B@1 is published. B@1 depends on A@1.
A@2 and B@2 are being considered for publication. A@2 has substantial
content changes compared to A@1. B@2 still depends on A, and is
otherwise completely identical to B@1.
In the previous scheme, B@2 would be published with a dependency on A@2.
I think that's a flaw. B@2 shouldn't be published. A@2 satisfies B@1's
dependency on A, and B@2 hasn't changed in any way to depend on the
specific changes in A@2 (otherwise the content that B@2 delivers must
have changed).
Here are the the new steps:
1) Do pkdep generate on all manifests. This step is unchanged from
current practice or the previous proposal.
2) Do pkdep resolve -P. This is unchanged from the previous proposal.
3) Use pkgsend publish-all to conditionally publish all packages. (I've
included the rough addition to the manpage that we'd make below.) The
steps to "conditionally publish all packages" are described in 4-7.
4) Collect the name and version of every package to be published.
5) Find the correct version of the package to compare against and
retrieve its manifest (this could be delayed till later, or all manifest
retrievals aggregated, but eventually we're going to need all the
manifests we're comparing against).
5.1) If the package name is incorporated by any package given to -i,
then search, first the destination repo, and then the reference repos in
order until a package with that name that matches the incorporate
dependency is found. If multiple versions of the package in that
repository match, take the latest one. If the package found is a
successor to the package being published, that's an error and
publication stops.
5.2) If the package name is not incorporated by an package given to -i,
then search the destination and then each reference repository until a
package is found which has the same name as the package being published
and a version that's less than or equal to the version of the package
being published. If multiple packages in the repository satisfy that
criteria, choose the latest one.
6) For each manifest:
6.1) Fill in hashes and other attributes related to file content.
6.2) Compare the manifest of the package being published with the
manifest of the package ignoring the specified actions (see 6.2.X). If
at least one relevant action is different, the manifest goes in the
"different set", otherwise it goes in the "identical set".
6.2.1) Attribute actions with the name pkg.fmri
6.2.2) Signature actions
6.2.3) Depend actions where the only difference is that in the manifest
to be published, the version is '@current' and in the manifest being
compared against, the version less than or equal to the version of the
package that that package is being compared against.
7) For each manifest in the different set:
7.1) Replace each <pkg_name>@current with the correct version.
7.1.1) If pkg_name is in the different set, then replace @current with
the version of that package that's going to be published.
7.1.2) If pkg_name isn't in the different set, then replace @current
with the version of that package that was being compared against.
7.2) Collapse and coalesce depend actions.
7.3) Publish the package
Rough spec of the command, please ignore formating and poor wording.
pkgsend publish-all [-r <reference_repo> ...] -s destination_repo [-d
source ...] [-b bundle ...] [-T pattern] [-i pkg_fmri ...] [-f file ...]
-v path_to_manifest ...
-s, -d, -b, and -T are exactly the same as the options to pkgsend publish.
-v With -v print to stdout a list of all the packages that were
published and all the packages that were omitted from publication.
-r These options create an ordered list of repositories which will be
used as sources to compare the packages to be published against. Each
repository will be searched in order until a package is found that
"matches". The first package found will be used, even if a package in a
later repository also satisfies the matches and was published later.
-i These options create an ordered list of incorporations to compare the
packages being published against. If package A is being published, then
the first package given as an option to i which incorporates A will be
used as the restriction. If none of the packages used as -i options
incorporates A, then the package "immediately less" than the version A
is going to be published at will be used. (Please see previous long
email exchange for the definition of "immediately less than".)
Path_to_manifest. Unlike pkgsend publish which merges all manifests
provided to produce a single package, each manifest provided must
completely define a single package. Passing N manifests causes N
packages to be published.
The following options are ideas about how to handle the fact that we
might want to pass one or two thousand paths to the command at one time.
-f This option takes a file which contains paths to manifests. This
option because a large enough number of packages might need to be
published at a single time that we'll run into shell issues. Each line
in the file is expected to be a path to a manifest and will be treated
identically to any paths passed on the command line.
--directory This option takes a path to a directory, assumes every
file/link inside that directory points to a manifest and attempts to
publish all of them.
Thanks for taking the time to review this,
Brock
On 03/05/12 12:00, Brock Pytlik wrote:
As part of Danek's design for the publication changes, we need a way
to defer dependency resolution until it's known whether a new version
of a package is going to be published or not. I've been struggling
with how to do it for a little while, but I think I've got things
nailed down now. What follows is a rough idea of how I envision the
publication process happening. I've written this as if the processing
was being done on the client side of things, but from step 3 on, it
could be happening on the server side.
1) Do pkgdep generate on all manifests to be published. This step is
unchanged from the present.
2) Do pkgdep resolve -P. -P is a new option which means "postpone
dependency versions." It also eliminates the dependency collapsing and
coalescing which currently happens during the resolve phase.
Dependencies inferred on system packages during this phase will have
versions attached to them, but those inferred on other packages also
being resolved will have versions of "@current."
3) Use pkgsend (in some possibly new form) to fill in file hashes and
possibly pkgmog to apply other transformations so that the all the
actions except for dependency actions, signature actions, and
attribute actions with the name pkg.fmri are what they would be when
the package is finally published. This step may or may not move file
data across to the repo.
4) Grab the repo lock. I'm being deliberately vague here. Since we're
comparing the manifest we're considering publishing with one in the
repository, if another package got published after we compared, but
before we published, we might make incorrect decisions about whether
to publish a package. This could be as simple as a convention of not
allowing multiple people to publish to the same repository at the same
time. It could be accomplished by comparing the catalog at comparison
time with the catalog when the packages are being published to ensure
the state of the world is what's expected. It could be an actual lock
on publishing to the repository.
5) For each manifest to be published compare all actions except
attributes whose name is pkg.fmri, signature actions (though if the
signing cert is different, might want to mark it as being different
anyway), and dependency actions with the version of the package
immediately less than the version to be published. Put all manifests
with at least one different action in the "different manifests" set.
Put the rest in the "identical manifests" set.
6) For all depend actions in manifests to be published which use
<pkg-name>@current in the target or predicate, if <pkg-name> is in the
set of "different manifests", replace @current with the version of the
package in the package to be published, otherwise replace @current
with the version of the previously published manifest
7) Now that all dependencies have fixed versions, collapse and
coalesce the dependencies for all the packages to be published.
8) For each manifest to be published, compare all actions except
attributes whose name is pkg.fmri and signature actions with the
version of the package immediately less than the version to be
published. Put all manifests which are different in the "actually to
be published" set.
9) Publish each manifest in the "actually to be published" set.
10) Release the repo lock.
I believe this approach will work. Please let me know if you see any
issues with the proposal.
Thanks,
Brock
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
