Re: [rust-dev] rustpkg use scenarios

[Graydon Hoare]

On 13-06-27 02:39 PM, SiegeLord wrote:

I've been trying to understand how rustpkg is meant to be used, and I
found that it didn't quite mesh with my experience with other package
management systems. Since it's quite incomplete and under-documented,
instead of trying to comment on the status quo, I decided its best to
form a few scenarios of usage, and hopefully have the people who
understand where rustpkg is going help me to see if these scenarios are
going to be possible (or if not, what's wrong with them). To that end,
the syntax for rustpkg will be my own and not the current syntax.


Rustpkg does have some documentation. It's sparse and additions would 
certainly be welcome. I suggest reading over the existing docs and 
discussions:


https://github.com/mozilla/rust/blob/master/doc/rustpkg.md
https://github.com/mozilla/rust/wiki/Bikeshed-rustpkg
https://mail.mozilla.org/pipermail/rust-dev/2013-February/003200.html

I'll try to answer your preludes and scenarios based on what I know of 
it, but please understand that there are a _lot_ of competing tensions 
on rustpkg and not every user's intuitions are immediately met on first 
encounter. Rather than redesign it (again) at this late a phase, I'd 
like to see it completed and deployed, and adjusted as-needed based on 
experience.



Prelude 1:

1 Package != 1 crate. I think a package should be able to contain
several crates. E.g. Debian packages routinely contain multiple
binaries/libraries etc.


This is true in the design we're currently working on.



Prelude 2:

Package source != package name. Regardless of where you get the package
from and no matter what the package file name was, the package metadata
(name/author/version) should not be affected, and should be the only
things that determine the identity of the package. I don't think these
things are inferrable, so I envision a metadata file with all those
things specified.


This is currently not true in the design we're working on. Or .. not 
exactly. Packages are given a pkgid, which is a URL-fragment-like name. 
It is _not_ a complete URL, but it's most of one, and it's standing in 
for having separate metadata. It is enough that:


  - It can be globally unambiguous by beginning with a DNS name
  - It can be locally overridden by prepending some local prefix
  - It may also be able to hint rustpkg at an internet access method

This does not mean that you must fetch a package named 
"github.com/graydon/foo" from github.com, but it means that if you don't 
have any other source for that package, you can guess at where to get 
it. And it has a unique name (assuming you decide to use that fact).


This was chosen very carefully, very intentionally, and (for the time 
being) we're not revisiting this choice. We experimented before with 
having multiple points of name-indirection or metadata and it appears to 
have just annoyed and confused everyone.



Scenario 1: Installation from a package file


I don't think this is on the current work list. It might be at some 
point. It won't be especially hard, but neither is it hard to just 
unpack a package in a workspace and build it using the existing 
directory-based approach.


It is not installing to /usr/local or any filesystem global location by 
default. It is intended to localize dependencies (in terms of using 
nearby directories) as much as possible by default.



Scenario 2: Installation from an appropriately structured directory


In these cases, you pass a pkgid and it finds an installation candidate 
using RUST_PATH. If it can't find a candidate it guesses an access 
method for a remote from the pkgid and pulls that into the first entry 
in RUST_PATH.



Scenario 3: Uninstallation


I don't think this is on the current work list but it's not hard.


Scenario 4: Fetching/building without installation


This is in the current work list. I do not think fetch is separate from 
build, but it might be. Tim will know more.



Scenario 5: Specifying a workspace


Workspaces (each of which contains subdirectories for sources, builds 
and installed binaries, per-package) are specified by RUST_PATH in the 
current design.


-Graydon

___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev

Re: [rust-dev] rustpkg use scenarios

[SiegeLord]

On 06/27/2013 06:14 PM, Graydon Hoare wrote:


This does not mean that you must fetch a package named
"github.com/graydon/foo" from github.com, but it means that if you don't
have any other source for that package, you can guess at where to get
it. And it has a unique name (assuming you decide to use that fact).


I have a rust project hosted on github, but the package name is 
certainly not github.com// in my mind. 
What if it was hosted on bitbucket? What if I delete the repository? I 
find it puzzling to have to place my local repository inside a directory 
named github.com// for me to be able to 
build my local copy using rustpkg.


It's the last bit that bothers me the most: I can't build my local 
project unless I introduce a completely irrelevant aspect (my git host 
and username) into my metadata (the directory structure).




This was chosen very carefully, very intentionally, and (for the time
being) we're not revisiting this choice. We experimented before with
having multiple points of name-indirection or metadata and it appears to
have just annoyed and confused everyone.


Most package system I've reviewed prior to writing that email used a 
metadata file in its package system. It really is not clear why Rust 
must be nearly unique in this point. In the 3 sources of documentation 
you've linked to there really wasn't a motivation behind why the 
widespread metadata file approach was wrong.


I understand that this is too late to change this, but at the same time 
I find integrating my (sole for now) Rust project into rustpkg to be 
really disruptive, relative to simply including a metadata file like 
I've done for my projects in other languages.


-SL
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev

Re: [rust-dev] rustpkg use scenarios

[Zack Corr]

I think you're confused here. You don't need to name the directory
github.com// at all, this is just an abstracted URL that can be
used to fetch a Rustpkg project from Github. More abstracted URLs could be
added to fetch projects from other sources.

Honestly, I agree with you. I still prefer requiring a central pkg.rs file
with a name, version and author defined. Sure, these can be inferred based
on this abstract URL concept, but it's nice having a central metadata file
that exists across sources. I'm still cloudy about how versioning is going
to be inferred based on branches and I think I will continue to be until
it's implemented and working well.

On Fri, Jun 28, 2013 at 11:34 AM, SiegeLord  wrote:

> On 06/27/2013 06:14 PM, Graydon Hoare wrote:
>
>  This does not mean that you must fetch a package named
>> "github.com/graydon/foo" from github.com, but it means that if you don't
>> have any other source for that package, you can guess at where to get
>> it. And it has a unique name (assuming you decide to use that fact).
>>
>
> I have a rust project hosted on github, but the package name is certainly
> not github.com//<**repository_name> in my mind. What if it
> was hosted on bitbucket? What if I delete the repository? I find it
> puzzling to have to place my local repository inside a directory named
> github.com//<**repository_name> for me to be able to build
> my local copy using rustpkg.
>
> It's the last bit that bothers me the most: I can't build my local project
> unless I introduce a completely irrelevant aspect (my git host and
> username) into my metadata (the directory structure).
>
>
>
>> This was chosen very carefully, very intentionally, and (for the time
>> being) we're not revisiting this choice. We experimented before with
>> having multiple points of name-indirection or metadata and it appears to
>> have just annoyed and confused everyone.
>>
>
> Most package system I've reviewed prior to writing that email used a
> metadata file in its package system. It really is not clear why Rust must
> be nearly unique in this point. In the 3 sources of documentation you've
> linked to there really wasn't a motivation behind why the widespread
> metadata file approach was wrong.
>
> I understand that this is too late to change this, but at the same time I
> find integrating my (sole for now) Rust project into rustpkg to be really
> disruptive, relative to simply including a metadata file like I've done for
> my projects in other languages.
>
> -SL
>
> __**_
> Rust-dev mailing list
> Rust-dev@mozilla.org
> https://mail.mozilla.org/**listinfo/rust-dev
>
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev

Re: [rust-dev] rustpkg use scenarios

[SiegeLord]

On 06/27/2013 10:09 PM, Zack Corr wrote:


I think you're confused here. You don't need to name the directory
github.com/ / at all, this is just an
abstracted URL that can be used to fetch a Rustpkg project from Github.
More abstracted URLs could be added to fetch projects from other sources.


Well, consider this. Let's say I have a project A has a different 
project B as its dependency, which I, by accident of history, host on 
github.


I.e. project A has this line in it (I think this is the proposed syntax):

extern mod "github.com/SiegeLord/ProjectB";

How would I get rustpkg to use my local copy of it? I assume it will try 
to do: rustpkg install github.com/SiegeLord/ProjectB which will only 
work the way I want if I re-create that directory structure on my system.


-SL
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev

Re: [rust-dev] rustpkg use scenarios

[Graydon Hoare]

On 13-06-27 06:34 PM, SiegeLord wrote:

On 06/27/2013 06:14 PM, Graydon Hoare wrote:


This does not mean that you must fetch a package named
"github.com/graydon/foo" from github.com, but it means that if you don't
have any other source for that package, you can guess at where to get
it. And it has a unique name (assuming you decide to use that fact).


I have a rust project hosted on github, but the package name is
certainly not github.com// in my mind.


That is what its pkgid would be, in the design we chose. It meets the 
relevant criteria:


  - decentralized
  - collision-free
  - "single string" package identification for use in 'extern mod'
  - support for remote and local packages, overrides, redirects


What if it was hosted on bitbucket?


Then you would either:

  - name it bitbucket.com/username/package
  - (re)name it something else, via the redirect procedures below


What if I delete the repository?


Then either you will:

  - redirect at server side[1]
  - have downstream users override or redirect locally
  - name it internally using an attribute, at the cost of losing the
ability for a user to fetch it remotely via pkgid


find it puzzling to have to place my local repository inside a directory
named github.com// for me to be able to
build my local copy using rustpkg.


It may be puzzling but it is both simple to explain and relatively 
robust against misconfiguration. In any case, you only need to do that 
if you expect to publish it. You can put it in a shorter named directory 
-- any name you like -- if you don't expect to.



It's the last bit that bothers me the most: I can't build my local
project unless I introduce a completely irrelevant aspect (my git host
and username) into my metadata (the directory structure).


When we did it the other way, people were upset that they had to make a 
separate file to hold metadata in, and they faced the difficulty of 
having no simple unique names to refer to packages by when denoting a 
dependency from one package to another or referring to a package on a 
command line. This structure reuses a single form of relatively robust 
structured identifiers for multiple roles (links in source code, remote 
access protocols, filesystem organization). It is economical, which is 
why we adopted it.



Most package system I've reviewed prior to writing that email used a
metadata file in its package system.


Which did you review, and which do you think we should imitate? Most 
package systems I reviewed have numerous other failure modes:


  - central index / namespace of all packages
  - shallow, collision-prone global namespace
  - poor to no integration with language module system
  - poor to no integration with revision control
  - weak or confusing support for mixing local overrides with remote
packages


It really is not clear why Rust  must be nearly unique in this point.


We made it quite clear during design that we were copying seemingly-wise 
and economical design choices from the Go package manager, and deviating 
in places where they were weak (eg. versioning)



In the 3 sources of documentation
you've linked to there really wasn't a motivation behind why the
widespread metadata file approach was wrong.


I suppose not, though I'd hoped it was obvious reading through the 
development threads. I hope I've outlined motives here well enough: that 
this approach appears to be more-economical with names and number of 
moving parts. Fewer layers of indirection, more reliance on constraints 
from the outside world.



I understand that this is too late to change this, but at the same time
I find integrating my (sole for now) Rust project into rustpkg to be
really disruptive, relative to simply including a metadata file like
I've done for my projects in other languages.


It is late in the process. I'm sorry that it's disappointing to you but 
to return to redesign at this point would mean shipping another whole 
development cycle without a working package manager.


I realize the design may not be identical to other systems, but I 
believe it's quite economical. We made it as simple as possible, so 
there were as few ways to get things wrong as possible. All you need to 
do is:


  - rename the package directory to match its pkgid
  - rename your crate entry points to {main,lib,test,bench}.rs

Is this really so disruptive? It's easy to explain and easy to get 
right, hard to get wrong. At least that's the idea.


-Graydon

[1] if it helps, we are also planning to offer a 
'pkg.rust-lang.org/user/foo' URL-redirection service, for people who 
"don't trust code hosting URLs". This is _exactly as reliable_ a 
mechanism as any other central-index thingy we might decide to operate.


___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev

Re: [rust-dev] rustpkg use scenarios

[Graydon Hoare]

On 13-06-27 07:55 PM, SiegeLord wrote:


Well, consider this. Let's say I have a project A has a different
project B as its dependency, which I, by accident of history, host on
github.

I.e. project A has this line in it (I think this is the proposed syntax):

extern mod "github.com/SiegeLord/ProjectB";

How would I get rustpkg to use my local copy of it?


You would make a local directory "$W/github.com/SiegeLord/ProjectB" in 
some workspace $W that's part of your RUST_PATH, that contains your 
local override.



I assume it will try
to do: rustpkg install github.com/SiegeLord/ProjectB which will only
work the way I want if I re-create that directory structure on my system.


Correct. I think Zack misunderstood your objection. To override a pkgid 
locally you would need to create a local directory of the same name.


-Graydon

___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev

Re: [rust-dev] rustpkg use scenarios

[Zack Corr]

Yeah, I misunderstood, apologies.

I was reading over the documents you linked earlier to see if there was any
further information on how versions were going to be handled. Surely
forcing packages without version control to be 0.1 is restrictive? I feel
as though the central metadata thing should still be optional, i.e. you can
supply a version number in the case that you will be distributing the
project as a tarball. It seems pkg.rs will be kept as an optional tool for
custom compilation logic, so it makes sense that the metadata could be
optional as well.


On Fri, Jun 28, 2013 at 1:30 PM, Graydon Hoare  wrote:

> On 13-06-27 07:55 PM, SiegeLord wrote:
>
>  Well, consider this. Let's say I have a project A has a different
>> project B as its dependency, which I, by accident of history, host on
>> github.
>>
>> I.e. project A has this line in it (I think this is the proposed syntax):
>>
>> extern mod "github.com/SiegeLord/ProjectB**";
>>
>> How would I get rustpkg to use my local copy of it?
>>
>
> You would make a local directory 
> "$W/github.com/SiegeLord/**ProjectB"
> in some workspace $W that's part of your RUST_PATH, that contains your
> local override.
>
>
>  I assume it will try
>> to do: rustpkg install github.com/SiegeLord/ProjectB which will only
>> work the way I want if I re-create that directory structure on my system.
>>
>
> Correct. I think Zack misunderstood your objection. To override a pkgid
> locally you would need to create a local directory of the same name.
>
> -Graydon
>
>
> __**_
> Rust-dev mailing list
> Rust-dev@mozilla.org
> https://mail.mozilla.org/**listinfo/rust-dev
>
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev

Re: [rust-dev] rustpkg use scenarios

[Zack Corr]

Quick thought: If you're going to definitely go along with inferring
everything, maybe you could look for the tarball or directory to be
optionally named something like "rust-sdl-v0.1.tar.gz" and extract the
version from that?


On Fri, Jun 28, 2013 at 2:52 PM, Zack Corr  wrote:

> Yeah, I misunderstood, apologies.
>
> I was reading over the documents you linked earlier to see if there was
> any further information on how versions were going to be handled. Surely
> forcing packages without version control to be 0.1 is restrictive? I feel
> as though the central metadata thing should still be optional, i.e. you can
> supply a version number in the case that you will be distributing the
> project as a tarball. It seems pkg.rs will be kept as an optional tool
> for custom compilation logic, so it makes sense that the metadata could be
> optional as well.
>
>
> On Fri, Jun 28, 2013 at 1:30 PM, Graydon Hoare wrote:
>
>> On 13-06-27 07:55 PM, SiegeLord wrote:
>>
>>  Well, consider this. Let's say I have a project A has a different
>>> project B as its dependency, which I, by accident of history, host on
>>> github.
>>>
>>> I.e. project A has this line in it (I think this is the proposed syntax):
>>>
>>> extern mod "github.com/SiegeLord/ProjectB**";
>>>
>>> How would I get rustpkg to use my local copy of it?
>>>
>>
>> You would make a local directory 
>> "$W/github.com/SiegeLord/**ProjectB"
>> in some workspace $W that's part of your RUST_PATH, that contains your
>> local override.
>>
>>
>>  I assume it will try
>>> to do: rustpkg install github.com/SiegeLord/ProjectB which will only
>>> work the way I want if I re-create that directory structure on my system.
>>>
>>
>> Correct. I think Zack misunderstood your objection. To override a pkgid
>> locally you would need to create a local directory of the same name.
>>
>> -Graydon
>>
>>
>> __**_
>> Rust-dev mailing list
>> Rust-dev@mozilla.org
>> https://mail.mozilla.org/**listinfo/rust-dev
>>
>
>
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev

Re: [rust-dev] rustpkg use scenarios

[Tim Chevalier]

Graydon answered this pretty thoroughly, but just to add:

On Thu, Jun 27, 2013 at 2:39 PM, SiegeLord  wrote:
> Scenario 4: Fetching/building without installation
>
> Building without installing:
>
>  rustpkg build 
>
> If you then run `rustpkg install ` or `rustpkg install --package bar`,
> then it'll use that pre-built copy without fetching the sources and building

This is the intended behavior, though currently it will rebuild,
because https://github.com/mozilla/rust/issues/7075 isn't implemented
yet.

> them again. Speaking of fetching, this would just fetch the sources:
>
>  rustpkg fetch 
>

Planned -- see https://github.com/mozilla/rust/issues/7242

Also, I opened https://github.com/mozilla/rust/issues/7447 for Zack's
suggestions about versions.

Cheers,
Tim

-- 
Tim Chevalier * http://catamorphism.org/ * Often in error, never in doubt
"Not a riot, it's a rebellion." -- Boots Riley
"Attention Bros and Trolls: When I call out your spew, I'm not angry,
I'm defiant." -- Reg Braithwaite
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev