[PATCH 02/23] midx: add midx format details to pack-format.txt
The multi-pack-index (MIDX) feature generalizes the existing pack- index (IDX) feature by indexing objects across multiple pack-files. Describe the basic file format, using a 12-byte header followed by a lookup table for a list of "chunks" which will be described later. The file ends with a footer containing a checksum using the hash algorithm. The header allows later versions to create breaking changes by advancing the version number. We can also change the hash algorithm using a different version value. We will add the individual chunk format information as we introduce the code that writes that information. Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 49 + 1 file changed, 49 insertions(+) diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt index 70a99fd142..17666b4bfc 100644 --- a/Documentation/technical/pack-format.txt +++ b/Documentation/technical/pack-format.txt @@ -252,3 +252,52 @@ Pack file entry: <+ corresponding packfile. 20-byte SHA-1-checksum of all of the above. + +== midx-*.midx files have the following format: + +The meta-index files refer to multiple pack-files and loose objects. + +In order to allow extensions that add extra data to the MIDX, we organize +the body into "chunks" and provide a lookup table at the beginning of the +body. The header includes certain length values, such as the number of packs, +the number of base MIDX files, hash lengths and types. + +All 4-byte numbers are in network order. + +HEADER: + + 4-byte signature: + The signature is: {'M', 'I', 'D', 'X'} + + 1-byte version number: + Git only writes or recognizes version 1 + + 1-byte Object Id Version + Git only writes or recognizes verion 1 (SHA-1) + + 1-byte number (C) of "chunks" + + 1-byte number (I) of base multi-pack-index files: + This value is currently always zero. + + 4-byte number (P) of pack files + +CHUNK LOOKUP: + + (C + 1) * 12 bytes providing the chunk offsets: + First 4 bytes describe chunk id. Value 0 is a terminating label. + Other 8 bytes provide offset in current file for chunk to start. + (Chunks are provided in file-order, so you can infer the length + using the next chunk position if necessary.) + + The remaining data in the body is described one chunk at a time, and + these chunks may be given in any order. Chunks are required unless + otherwise specified. + +CHUNK DATA: + + (This section intentionally left incomplete.) + +TRAILER: + + H-byte HASH-checksum of all of the above. -- 2.18.0.rc1
Re: [PATCH 02/23] midx: add midx format details to pack-format.txt
Hi Derrick, On Thu, Jun 7, 2018 at 7:03 AM Derrick Stolee wrote: > > The multi-pack-index (MIDX) feature generalizes the existing pack- > index (IDX) feature by indexing objects across multiple pack-files. > > Describe the basic file format, using a 12-byte header followed by > a lookup table for a list of "chunks" which will be described later. > The file ends with a footer containing a checksum using the hash > algorithm. > > The header allows later versions to create breaking changes by > advancing the version number. We can also change the hash algorithm > using a different version value. > > We will add the individual chunk format information as we introduce > the code that writes that information. > > Signed-off-by: Derrick Stolee > --- > Documentation/technical/pack-format.txt | 49 + > 1 file changed, 49 insertions(+) > > diff --git a/Documentation/technical/pack-format.txt > b/Documentation/technical/pack-format.txt > index 70a99fd142..17666b4bfc 100644 > --- a/Documentation/technical/pack-format.txt > +++ b/Documentation/technical/pack-format.txt > @@ -252,3 +252,52 @@ Pack file entry: <+ > corresponding packfile. > > 20-byte SHA-1-checksum of all of the above. > + > +== midx-*.midx files have the following format: > + > +The meta-index files refer to multiple pack-files and loose objects. So is it meta or multi? > +In order to allow extensions that add extra data to the MIDX, we organize > +the body into "chunks" and provide a lookup table at the beginning of the > +body. The header includes certain length values, such as the number of packs, > +the number of base MIDX files, hash lengths and types. > + > +All 4-byte numbers are in network order. > + > +HEADER: > + > + 4-byte signature: > + The signature is: {'M', 'I', 'D', 'X'} > + > + 1-byte version number: > + Git only writes or recognizes version 1 > + > + 1-byte Object Id Version > + Git only writes or recognizes verion 1 (SHA-1) s/verion/version/ > + 1-byte number (C) of "chunks" > + > + 1-byte number (I) of base multi-pack-index files: > + This value is currently always zero. Oh? Are meta-index and multi-index files different things? > + 4-byte number (P) of pack files > + > +CHUNK LOOKUP: > + > + (C + 1) * 12 bytes providing the chunk offsets: > + First 4 bytes describe chunk id. Value 0 is a terminating label. > + Other 8 bytes provide offset in current file for chunk to start. > + (Chunks are provided in file-order, so you can infer the length > + using the next chunk position if necessary.) It is so nice to have the header also have 12 bytes, so it fits right into the lookup table. So an alternative point of view: If a chunk needs to store more than 8 bytes, we'll have an offset after the first 4 bytes that describe the chunk, otherwise you can store the 8 bytes of information directly after the 4 bytes. "MIDX" is a special chunk and must come first (does it?) and only once as it contains the version number. > + The remaining data in the body is described one chunk at a time, and > + these chunks may be given in any order. Chunks are required unless > + otherwise specified. > + > +CHUNK DATA: > + > + (This section intentionally left incomplete.) > + > +TRAILER: > + > + H-byte HASH-checksum of all of the above. This means we have to rehash the whole file for updating its contents. okay.
Re: [PATCH 02/23] midx: add midx format details to pack-format.txt
On 6/11/2018 3:19 PM, Stefan Beller wrote: Hi Derrick, On Thu, Jun 7, 2018 at 7:03 AM Derrick Stolee wrote: The multi-pack-index (MIDX) feature generalizes the existing pack- index (IDX) feature by indexing objects across multiple pack-files. Describe the basic file format, using a 12-byte header followed by a lookup table for a list of "chunks" which will be described later. The file ends with a footer containing a checksum using the hash algorithm. The header allows later versions to create breaking changes by advancing the version number. We can also change the hash algorithm using a different version value. We will add the individual chunk format information as we introduce the code that writes that information. Signed-off-by: Derrick Stolee --- Documentation/technical/pack-format.txt | 49 + 1 file changed, 49 insertions(+) diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt index 70a99fd142..17666b4bfc 100644 --- a/Documentation/technical/pack-format.txt +++ b/Documentation/technical/pack-format.txt @@ -252,3 +252,52 @@ Pack file entry: <+ corresponding packfile. 20-byte SHA-1-checksum of all of the above. + +== midx-*.midx files have the following format: + +The meta-index files refer to multiple pack-files and loose objects. So is it meta or multi? Good catch. We were calling this the meta-index internally before changing to "multi-pack-index" (helps to not change the acronym). +In order to allow extensions that add extra data to the MIDX, we organize +the body into "chunks" and provide a lookup table at the beginning of the +body. The header includes certain length values, such as the number of packs, +the number of base MIDX files, hash lengths and types. + +All 4-byte numbers are in network order. + +HEADER: + + 4-byte signature: + The signature is: {'M', 'I', 'D', 'X'} + + 1-byte version number: + Git only writes or recognizes version 1 + + 1-byte Object Id Version + Git only writes or recognizes verion 1 (SHA-1) s/verion/version/ + 1-byte number (C) of "chunks" + + 1-byte number (I) of base multi-pack-index files: + This value is currently always zero. Oh? Are meta-index and multi-index files different things? Not intended to be different things, but this number is related to making the feature incremental. + 4-byte number (P) of pack files + +CHUNK LOOKUP: + + (C + 1) * 12 bytes providing the chunk offsets: + First 4 bytes describe chunk id. Value 0 is a terminating label. + Other 8 bytes provide offset in current file for chunk to start. + (Chunks are provided in file-order, so you can infer the length + using the next chunk position if necessary.) It is so nice to have the header also have 12 bytes, so it fits right into the lookup table. So an alternative point of view: If a chunk needs to store more than 8 bytes, we'll have an offset after the first 4 bytes that describe the chunk, otherwise you can store the 8 bytes of information directly after the 4 bytes. "MIDX" is a special chunk and must come first (does it?) and only once as it contains the version number. This sounds feasible, but unnecessarily complicated. I don't think any other chunk will be this small. + The remaining data in the body is described one chunk at a time, and + these chunks may be given in any order. Chunks are required unless + otherwise specified. + +CHUNK DATA: + + (This section intentionally left incomplete.) + +TRAILER: + + H-byte HASH-checksum of all of the above. This means we have to rehash the whole file for updating its contents. okay.
Re: [PATCH 02/23] midx: add midx format details to pack-format.txt
> >> + (C + 1) * 12 bytes providing the chunk offsets: > >> + First 4 bytes describe chunk id. Value 0 is a terminating > >> label. > >> + Other 8 bytes provide offset in current file for chunk to > >> start. > >> + (Chunks are provided in file-order, so you can infer the length > >> + using the next chunk position if necessary.) > > It is so nice to have the header also have 12 bytes, so it fits right into > > the > > lookup table. So an alternative point of view: > > > >If a chunk needs to store more than 8 bytes, we'll have an offset after > >the first 4 bytes that describe the chunk, otherwise you can store the 8 > > bytes > >of information directly after the 4 bytes. > > "MIDX" is a special chunk and must come first (does it?) and only once > >as it contains the version number. > > This sounds feasible, but unnecessarily complicated. I don't think any > other chunk will be this small. I was just writing it as a way to test if I really understood what you said in the doc, not as a suggestion to incorporate it. Stefan