Hello,
Here's my draft proposal for extending the directory protocol to
support compression scheme negotiation using the semantics of the HTTP
protocol. This is part of the work that Nick and I are looking into
for our Sponsor4 design.
All feedback is highly appreciated :-)
Cheers,
Alex.
Filename: xxx-directory-compression-scheme-negotiation.txt
Title: Directory Compression Scheme Negotiation
Author: Alexander Færøy
Created: 2017-03-06
Status: Draft
Target: N/A
0. Overview
This document describes a method to provide and use different
compression schemes in Tor's directory specification[0] and let it be
up the client and server to negotiate a mutually supported scheme
using the semantics of the HTTP protocol.
Furthermore this proposal also extends Tor's directory protocol with
support for the LZMA2 and Zstandard compression schemes.
1. Motivation
Currently Tor serves each directory client with its different document
flavours in either an uncompressed format or, if the client adds a
".z"-suffix to the URL file path, a zlib-compressed document.
This have historically been non-problematic, but it disallows us from
easily extending the set of supported compression schemes.
Some of the problems this proposal is trying to aid:
- We currently only support zlib-based compression schemes and there
is no way for directory servers or clients to announce which
compression schemes they support. Zlib might not be the ideal
compression scheme for all purposes.
- It is not easily possible to add support for additional
compression schemes without adding additional file extensions or
flavours of the directory documents.
- In low-bandwidth and/or low-memory client scenarios it is useful
to be able to limit the amount of supported compression schemes to
have a client only support the most efficient compression scheme
for the given use-case and have the directory servers support the
most commonly available compression schemes used throughout the
network.
- We add support for the LZMA2 compression scheme, which yields
better compressed size and decompression time at the expensive of
higher compression time and higher memory usage.
- We add support for the Zstandard compression scheme, which yields
better compression ratio than GZip, but slightly worse than LZMA2,
but with a smaller CPU and memory footprint than LZMA2.
2. Analysis
We investigated the compression ratio, memory usage, memory allocation
strategies, and execution time for compression and decompression of
the GZip, BZip2, LZMA2, and Zstandard compression schemes at
compression levels 1 through 9.
The data used in this analysis can be found in [1] and the `bench`
tool for generating the data can be found in [2].
During the preparation for this proposal Nick have analysed
compressing consensus diffs using both GZip, LZMA2, and Zstandard. The
result of Nick's analysis can be found in [3].
We must continue to support both "gzip", "deflate", and "identity"
which are the currently available compression schemes in the Tor
network.
Further to enhance the compression ratio Nick have also worked on
proposal #274 (Rotate onion keys less frequently), #275 (Stop
including meaningful "published" time in microdescriptor consensus),
#276 (Report bandwidth with lower granularity in consensus documents),
and #277 (Detect multiple relay instances running with same ID) which
all aid in making our consensus documents less dynamic.
3. Proposal
We extend the directory client requests to include the
"Accept-Encoding" header as part of its request. The "Accept-Encoding"
header should contain a comma-separated list of names of the
compression schemes of which the client supports.
For example:
GET / HTTP/1.0
Accept-Encoding: zstd, xz, gzip, deflate
When a directory server receives a request with the "Accept-Encoding"
header included it must decide on a mutually supported compression
scheme and add the "Content-Encoding" header to its response and thus
notifying the client of its decision. The "Content-Encoding" header
can at most contain one supported compression scheme. If no mutual
compression scheme can be negotiated the server must respond with an
HTTP error status code of 415 "Unsupported Media Type".
For example:
HTTP/1.0 200 OK
Content-Length: 1337
Connection: close
Content-Encoding: zstd
Currently supported compression scheme names includes "identity",
"gzip", and "deflate". This proposal adds two additional compression
scheme named "xz" (LZMA2) and "zstd" (Zstandard).
All compression scheme names are case-insensitive.
The "deflate", "gzip", and "identity" compression schemes must be
supported by directory servers for backwards compatibility.
Additionally, when a client, that supports this proposals, makes a
request to a
