Re: [pkg-discuss] on-disk format proposal v2

2010-05-25 Thread Stephen Hahn
* Shawn Walker [2010-05-24 18:31]: > On 05/18/10 06:00 PM, Stephen Hahn wrote: > >* Shawn Walker [2010-05-18 15:43]: > ... > >>Could you expound a bit on your possible concerns here and/or what > >>you mean by "deliver the origin(s)"? > > > > "Deliver the origin(s)" means that the origin of eac

Re: [pkg-discuss] on-disk format proposal v2

2010-05-24 Thread Shawn Walker
On 05/18/10 06:00 PM, Stephen Hahn wrote: * Shawn Walker [2010-05-18 15:43]: ... Could you expound a bit on your possible concerns here and/or what you mean by "deliver the origin(s)"? "Deliver the origin(s)" means that the origin of each publisher in the .p5p is provided in the corres

Re: [pkg-discuss] on-disk format proposal v2

2010-05-20 Thread Shawn Walker
On 05/20/10 03:15 PM, Glynn Foster wrote: On 19/05/2010, at 8:38 AM, Shawn Walker wrote: Greetings, Attached is the 2nd draft for the addition of an on-disk format to pkg(5). Any and all feedback is greatly appreciated by the end of May 26th, 2010. For those that saw the first draft, you can

Re: [pkg-discuss] on-disk format proposal v2

2010-05-20 Thread Glynn Foster
On 19/05/2010, at 8:38 AM, Shawn Walker wrote: Greetings, Attached is the 2nd draft for the addition of an on-disk format to pkg(5). Any and all feedback is greatly appreciated by the end of May 26th, 2010. For those that saw the first draft, you can find a webrev with differences her

Re: [pkg-discuss] on-disk format proposal v2

2010-05-20 Thread Darren J Moffat
On 19/05/2010 21:11, Shawn Walker wrote: If archive integrity verification is needed, (beyond simple accidental protection), my inclination would be to simply sign the archive using gpg and provide the resulting '.asc' file alongside the archive. That's a fairly standard, accepted practice as far

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread Stephen Hahn
* Shawn Walker [2010-05-19 15:11]: > On 05/19/10 04:57 PM, Stephen Hahn wrote: > >* Shawn Walker [2010-05-19 14:42]: > >>On 05/18/10 06:00 PM, Stephen Hahn wrote: > >>>* Shawn Walker [2010-05-18 15:43]: > >>... > Could you be more explicit about what you mean by this? I don't > quite c

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread Shawn Walker
On 05/19/10 04:57 PM, Stephen Hahn wrote: * Shawn Walker [2010-05-19 14:42]: On 05/18/10 06:00 PM, Stephen Hahn wrote: * Shawn Walker [2010-05-18 15:43]: ... Could you be more explicit about what you mean by this? I don't quite connect this to section 5.1's text. I was hoping that

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread Stephen Hahn
* Shawn Walker [2010-05-19 14:42]: > On 05/18/10 06:00 PM, Stephen Hahn wrote: > >* Shawn Walker [2010-05-18 15:43]: > ... > >>Could you be more explicit about what you mean by this? I don't > >>quite connect this to section 5.1's text. > > > > I was hoping that > > > > pkg install -g http:/

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread Shawn Walker
On 05/18/10 06:00 PM, Stephen Hahn wrote: * Shawn Walker [2010-05-18 15:43]: ... Could you be more explicit about what you mean by this? I don't quite connect this to section 5.1's text. I was hoping that pkg install -g http://example.com/foo.p5p would work. To clearly set expe

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread Shawn Walker
On 05/19/10 03:56 PM, Jason King wrote: On Tue, May 18, 2010 at 3:38 PM, Shawn Walker wrote: Greetings, Attached is the 2nd draft for the addition of an on-disk format to pkg(5). Any and all feedback is greatly appreciated by the end of May 26th, 2010. For those that saw the first draft, yo

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread Jason King
On Tue, May 18, 2010 at 3:38 PM, Shawn Walker wrote: > Greetings, > > Attached is the 2nd draft for the addition of an on-disk format to pkg(5). >  Any and all feedback is greatly appreciated by the end of May 26th, 2010. > > For those that saw the first draft, you can find a webrev with differenc

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread Shawn Walker
On 05/19/10 11:49 AM, Darren J Moffat wrote: On 19/05/2010 17:36, Shawn Walker wrote: On 05/19/10 08:14 AM, Darren J Moffat wrote: First a general comment, I'm really impressed with how "simple" this actually looks particularly given you have been able to use an existing archive format. I assu

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread Stephen Hahn
* Shawn Walker [2010-05-18 18:19]: > On 05/18/10 06:00 PM, Stephen Hahn wrote: > >* Shawn Walker [2010-05-18 15:43]: > >>On 05/18/10 04:57 PM, Stephen Hahn wrote: > >>> 4. (5.4) Using a .p5p archive as an origin is... unusual. I would > >>> have expected the .p5p to deliver the origin(

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread johansen
On Tue, May 18, 2010 at 08:32:32PM -0500, Shawn Walker wrote: > I would note that Apache's mod_cache and mod_disk_cache appear to > properly handle byte-range requests. Ok, cool. That's good to know. > > In the case where we're performing parallel file requests, we can > > spread the I/O out

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread Darren J Moffat
On 19/05/2010 17:36, Shawn Walker wrote: On 05/19/10 08:14 AM, Darren J Moffat wrote: First a general comment, I'm really impressed with how "simple" this actually looks particularly given you have been able to use an existing archive format. I assume when you mention ZIP as a format that was c

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread Shawn Walker
On 05/19/10 10:56 AM, Nicolas Williams wrote: On Wed, May 19, 2010 at 04:33:52PM +0100, Darren J Moffat wrote: ... Maybe nothing if we assume the package contents themselves are signed - and if they aren't I don't see any benefit in doing anything additional in the on-disk repo format. It may

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread Shawn Walker
On 05/19/10 08:14 AM, Darren J Moffat wrote: First a general comment, I'm really impressed with how "simple" this actually looks particularly given you have been able to use an existing archive format. I assume when you mention ZIP as a format that was considered the JAR variant of it was too.

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread Nicolas Williams
On Wed, May 19, 2010 at 04:33:52PM +0100, Darren J Moffat wrote: > On 19/05/2010 16:23, Nicolas Williams wrote: > >Integrity protection here could best be handled by ZFS, and by using a > >snapshot to access the on-disk repo. Granted, that would mean that > >IPS on systems that don't support ZFS w

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread Darren J Moffat
On 19/05/2010 16:23, Nicolas Williams wrote: On Wed, May 19, 2010 at 02:14:40PM +0100, Darren J Moffat wrote: I really have only one comment and that is about integrity protection of the on disk format. [...] Is there really a risk here ? Compare this to what ZFS does. It uses a Merkle tree

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread Nicolas Williams
On Wed, May 19, 2010 at 02:14:40PM +0100, Darren J Moffat wrote: > I really have only one comment and that is about integrity > protection of the on disk format. > > [...] > > Is there really a risk here ? > > Compare this to what ZFS does. It uses a Merkle tree of checksums > going all they wa

Re: [pkg-discuss] on-disk format proposal v2

2010-05-19 Thread Darren J Moffat
First a general comment, I'm really impressed with how "simple" this actually looks particularly given you have been able to use an existing archive format. I assume when you mention ZIP as a format that was considered the JAR variant of it was too. I really have only one comment and that is

Re: [pkg-discuss] on-disk format proposal v2

2010-05-18 Thread Shawn Walker
On 05/18/10 07:30 PM, johan...@opensolaris.org wrote: On Tue, May 18, 2010 at 06:26:11PM -0500, Shawn Walker wrote: On 05/18/10 04:33 PM, johan...@opensolaris.org wrote: I have to confess that there's not much of a use case for this type of transport access. Remote selective extraction is esse

Re: [pkg-discuss] on-disk format proposal v2

2010-05-18 Thread Shawn Walker
On 05/18/10 06:00 PM, Stephen Hahn wrote: * Shawn Walker [2010-05-18 15:43]: On 05/18/10 04:57 PM, Stephen Hahn wrote: Thanks for a thorough write-up. A first batch of questions and comments: 1. Is a .p5i present for all publishers in a v2 repository? Why or why not? I d

Re: [pkg-discuss] on-disk format proposal v2

2010-05-18 Thread johansen
On Tue, May 18, 2010 at 06:26:11PM -0500, Shawn Walker wrote: > On 05/18/10 04:33 PM, johan...@opensolaris.org wrote: > >I have to confess that there's not much of a use case for this type of > >transport access. Remote selective extraction is essentially what we > >have now through the file/0 int

Re: [pkg-discuss] on-disk format proposal v2

2010-05-18 Thread Shawn Walker
On 05/18/10 04:33 PM, johan...@opensolaris.org wrote: On Tue, May 18, 2010 at 04:20:12PM -0500, Shawn Walker wrote: Realistically, I don't expect this to be used that often. For optimal access to the archive, there should really only be one index file (especially if you were performing remote,

Re: [pkg-discuss] on-disk format proposal v2

2010-05-18 Thread Stephen Hahn
* Shawn Walker [2010-05-18 15:43]: > On 05/18/10 04:57 PM, Stephen Hahn wrote: > > > > Thanks for a thorough write-up. A first batch of questions and > > comments: > > > > 1. Is a .p5i present for all publishers in a v2 repository? Why or > > why not? > > I didn't make it required

Re: [pkg-discuss] on-disk format proposal v2

2010-05-18 Thread Shawn Walker
On 05/18/10 04:57 PM, Stephen Hahn wrote: Thanks for a thorough write-up. A first batch of questions and comments: 1. Is a .p5i present for all publishers in a v2 repository? Why or why not? I didn't make it required simply to minimise configuration hassle. Since we can't

Re: [pkg-discuss] on-disk format proposal v2

2010-05-18 Thread johansen
On Tue, May 18, 2010 at 04:20:12PM -0500, Shawn Walker wrote: > Realistically, I don't expect this to be used that often. For > optimal access to the archive, there should really only be one index > file (especially if you were performing remote, selective > extraction). I have to confess that th

Re: [pkg-discuss] on-disk format proposal v2

2010-05-18 Thread Shawn Walker
On 05/18/10 04:11 PM, johan...@opensolaris.org wrote: On Tue, May 18, 2010 at 03:38:47PM -0500, Shawn Walker wrote: ... 4.3.8. Package Archive Index Specification: The pkg(5) archive index file enables fast, efficient access to the contents of an archive. It

Re: [pkg-discuss] on-disk format proposal v2

2010-05-18 Thread johansen
On Tue, May 18, 2010 at 03:38:47PM -0500, Shawn Walker wrote: ... > 4.3.8. Package Archive Index Specification: > > The pkg(5) archive index file enables fast, efficient access > to the contents of an archive. It contains an entry for all > files in th

[pkg-discuss] on-disk format proposal v2

2010-05-18 Thread Shawn Walker
Greetings, Attached is the 2nd draft for the addition of an on-disk format to pkg(5). Any and all feedback is greatly appreciated by the end of May 26th, 2010. For those that saw the first draft, you can find a webrev with differences here: http://cr.opensolaris.org/~swalker/pkg-ondisk-