joerg.sonnenberger updated this revision to Diff 4932. joerg.sonnenberger retitled this revision from "wireproto: support for pullbundles" to "* wireproto: support for pullbundles".
REPOSITORY rHG Mercurial CHANGES SINCE LAST UPDATE https://phab.mercurial-scm.org/D1856?vs=4853&id=4932 REVISION DETAIL https://phab.mercurial-scm.org/D1856 AFFECTED FILES hgext/clonebundles.py mercurial/bundle2.py mercurial/commands.py mercurial/configitems.py mercurial/exchange.py mercurial/help/config.txt mercurial/wireproto.py tests/common-pattern.py tests/test-acl.t tests/test-bookmarks-pushpull.t tests/test-debugcommands.t tests/test-help.t tests/test-http-bad-server.t tests/test-pull-bundle.t tests/test-ssh-bundle1.t tests/test-ssh.t CHANGE DETAILS diff --git a/tests/test-ssh.t b/tests/test-ssh.t --- a/tests/test-ssh.t +++ b/tests/test-ssh.t @@ -486,7 +486,7 @@ devel-peer-request: between devel-peer-request: pairs: 81 bytes sending between command - remote: 384 + remote: 399 remote: capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS$ unbundle=HG10GZ,HG10BZ,HG10UN remote: 1 query 1; heads diff --git a/tests/test-ssh-bundle1.t b/tests/test-ssh-bundle1.t --- a/tests/test-ssh-bundle1.t +++ b/tests/test-ssh-bundle1.t @@ -467,7 +467,7 @@ running .* ".*/dummyssh" ['"]user@dummy['"] ('|")hg -R remote serve --stdio('|") (re) sending hello command sending between command - remote: 384 + remote: 399 remote: capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS$ unbundle=HG10GZ,HG10BZ,HG10UN remote: 1 preparing listkeys for "bookmarks" diff --git a/tests/test-pull-bundle.t b/tests/test-pull-bundle.t new file mode 100644 --- /dev/null +++ b/tests/test-pull-bundle.t @@ -0,0 +1,155 @@ + $ hg init repo + $ cd repo + $ echo foo > foo + $ hg ci -qAm 'add foo' + $ echo >> foo + $ hg ci -m 'change foo' + $ hg up -qC 0 + $ echo bar > bar + $ hg ci -qAm 'add bar' + + $ hg log + changeset: 2:effea6de0384 + tag: tip + parent: 0:bbd179dfa0a7 + user: test + date: Thu Jan 01 00:00:00 1970 +0000 + summary: add bar + + changeset: 1:ed1b79f46b9a + user: test + date: Thu Jan 01 00:00:00 1970 +0000 + summary: change foo + + changeset: 0:bbd179dfa0a7 + user: test + date: Thu Jan 01 00:00:00 1970 +0000 + summary: add foo + + $ cd .. + +Test pullbundle functionality + + $ cd repo + $ cat <<EOF > .hg/hgrc + > [server] + > pullbundle = True + > [extensions] + > blackbox = + > EOF + $ hg bundle --base null -r 0 .hg/0.hg + 1 changesets found + $ hg bundle --base 0 -r 1 .hg/1.hg + 1 changesets found + $ hg bundle --base 1 -r 2 .hg/2.hg + 1 changesets found + $ cat <<EOF > .hg/pullbundles.manifest + > 2.hg heads=effea6de0384e684f44435651cb7bd70b8735bd4 bases=bbd179dfa0a71671c253b3ae0aa1513b60d199fa + > 1.hg heads=ed1b79f46b9a29f5a6efa59cf12fcfca43bead5a bases=bbd179dfa0a71671c253b3ae0aa1513b60d199fa + > 0.hg heads=bbd179dfa0a71671c253b3ae0aa1513b60d199fa + > EOF + $ hg --config blackbox.track=debug --debug serve -p $HGPORT2 -d --pid-file=../repo.pid + listening at http://localhost:$HGPORT2/ (bound to $LOCALIP:$HGPORT2) + $ cat ../repo.pid >> $DAEMON_PIDS + $ cd .. + $ hg clone -r 0 http://localhost:$HGPORT2/ repo.pullbundle + adding changesets + adding manifests + adding file changes + added 1 changesets with 1 changes to 1 files + new changesets bbd179dfa0a7 + updating to branch default + 1 files updated, 0 files merged, 0 files removed, 0 files unresolved + $ cd repo.pullbundle + $ hg pull -r 1 + pulling from http://localhost:$HGPORT2/ + searching for changes + adding changesets + adding manifests + adding file changes + added 1 changesets with 1 changes to 1 files + new changesets ed1b79f46b9a + (run 'hg update' to get a working copy) + $ hg pull -r 2 + pulling from http://localhost:$HGPORT2/ + searching for changes + adding changesets + adding manifests + adding file changes + added 1 changesets with 1 changes to 1 files (+1 heads) + new changesets effea6de0384 + (run 'hg heads' to see heads, 'hg merge' to merge) + $ cd .. + $ killdaemons.py + $ grep 'sending pullbundle ' repo/.hg/blackbox.log + * sending pullbundle "0.hg" (glob) + * sending pullbundle "1.hg" (glob) + * sending pullbundle "2.hg" (glob) + $ rm repo/.hg/blackbox.log + +Test pullbundle functionality for incremental pulls + + $ cd repo + $ hg --config blackbox.track=debug --debug serve -p $HGPORT2 -d --pid-file=../repo.pid + listening at http://localhost:$HGPORT2/ (bound to $LOCALIP:$HGPORT2) + $ cat ../repo.pid >> $DAEMON_PIDS + $ cd .. + $ hg clone http://localhost:$HGPORT2/ repo.pullbundle2 + requesting all changes + adding changesets + adding manifests + adding file changes + added 1 changesets with 1 changes to 1 files + adding changesets + adding manifests + adding file changes + added 1 changesets with 1 changes to 1 files + adding changesets + adding manifests + adding file changes + added 1 changesets with 1 changes to 1 files (+1 heads) + new changesets bbd179dfa0a7:ed1b79f46b9a + updating to branch default + 1 files updated, 0 files merged, 0 files removed, 0 files unresolved + $ killdaemons.py + $ grep 'sending pullbundle ' repo/.hg/blackbox.log + * sending pullbundle "0.hg" (glob) + * sending pullbundle "2.hg" (glob) + * sending pullbundle "1.hg" (glob) + $ rm repo/.hg/blackbox.log + +Test recovery from misconfigured server sending no new data + + $ cd repo + $ cat <<EOF > .hg/pullbundles.manifest + > 0.hg heads=ed1b79f46b9a29f5a6efa59cf12fcfca43bead5a bases=bbd179dfa0a71671c253b3ae0aa1513b60d199fa + > 0.hg heads=bbd179dfa0a71671c253b3ae0aa1513b60d199fa + > EOF + $ hg --config blackbox.track=debug --debug serve -p $HGPORT2 -d --pid-file=../repo.pid + listening at http://localhost:$HGPORT2/ (bound to $LOCALIP:$HGPORT2) + $ cat ../repo.pid >> $DAEMON_PIDS + $ cd .. + $ hg clone -r 0 http://localhost:$HGPORT2/ repo.pullbundle3 + adding changesets + adding manifests + adding file changes + added 1 changesets with 1 changes to 1 files + new changesets bbd179dfa0a7 + updating to branch default + 1 files updated, 0 files merged, 0 files removed, 0 files unresolved + $ cd repo.pullbundle3 + $ hg pull -r 1 + pulling from http://localhost:$HGPORT2/ + searching for changes + adding changesets + adding manifests + adding file changes + added 0 changesets with 0 changes to 1 files + abort: 00changelog.i@ed1b79f46b9a: no node! + [255] + $ cd .. + $ killdaemons.py + $ grep 'sending pullbundle ' repo/.hg/blackbox.log + * sending pullbundle "0.hg" (glob) + * sending pullbundle "0.hg" (glob) + $ rm repo/.hg/blackbox.log diff --git a/tests/test-http-bad-server.t b/tests/test-http-bad-server.t --- a/tests/test-http-bad-server.t +++ b/tests/test-http-bad-server.t @@ -118,9 +118,9 @@ write(23) -> Server: badhttpserver\r\n write(37) -> Date: Fri, 14 Apr 2017 00:00:00 GMT\r\n write(41) -> Content-Type: application/mercurial-0.1\r\n - write(21) -> Content-Length: 417\r\n + write(21) -> Content-Length: 432\r\n write(2) -> \r\n - write(417) -> lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 bundle2=HG20%0Abookmarks%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=none + write(432) -> lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_NO_PHASES_CAPS$ unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=none readline(4? from 65537) -> (26) GET /?cmd=batch HTTP/1.1\r\n (glob) readline(1? from -1) -> (1?) Accept-Encoding* (glob) read limit reached; closing socket @@ -159,9 +159,9 @@ write(23) -> Server: badhttpserver\r\n write(37) -> Date: Fri, 14 Apr 2017 00:00:00 GMT\r\n write(41) -> Content-Type: application/mercurial-0.1\r\n - write(21) -> Content-Length: 417\r\n + write(21) -> Content-Length: 432\r\n write(2) -> \r\n - write(417) -> lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 bundle2=HG20%0Abookmarks%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=none + write(432) -> lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_NO_PHASES_CAPS$ unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=none readline(13? from 65537) -> (26) GET /?cmd=batch HTTP/1.1\r\n (glob) readline(1?? from -1) -> (27) Accept-Encoding: identity\r\n (glob) readline(8? from -1) -> (29) vary: X-HgArg-1,X-HgProto-1\r\n (glob) @@ -216,9 +216,9 @@ write(23) -> Server: badhttpserver\r\n write(37) -> Date: Fri, 14 Apr 2017 00:00:00 GMT\r\n write(41) -> Content-Type: application/mercurial-0.1\r\n - write(21) -> Content-Length: 430\r\n + write(21) -> Content-Length: 445\r\n write(2) -> \r\n - write(430) -> lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 bundle2=HG20%0Abookmarks%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httppostargs httpmediatype=0.1rx,0.1tx,0.2tx compression=none + write(445) -> lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_NO_PHASES_CAPS$ unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httppostargs httpmediatype=0.1rx,0.1tx,0.2tx compression=none readline\(14[67] from 65537\) -> \(2[67]\) POST /\?cmd=batch HTTP/1.1\\r\\n (re) readline\(1(19|20) from -1\) -> \(27\) Accept-Encoding: identity\\r\\n (re) readline(9? from -1) -> (41) content-type: application/mercurial-0.1\r\n (glob) @@ -275,7 +275,7 @@ $ cat hg.pid > $DAEMON_PIDS $ hg clone http://localhost:$HGPORT/ clone - abort: HTTP request error (incomplete response; expected 397 bytes got 20) + abort: HTTP request error (incomplete response; expected 412 bytes got 20) (this may be an intermittent network failure; if the error persists, consider contacting the network or server operator) [255] @@ -292,16 +292,16 @@ write(23 from 23) -> (121) Server: badhttpserver\r\n write(37 from 37) -> (84) Date: Fri, 14 Apr 2017 00:00:00 GMT\r\n write(41 from 41) -> (43) Content-Type: application/mercurial-0.1\r\n - write(21 from 21) -> (22) Content-Length: 417\r\n + write(21 from 21) -> (22) Content-Length: 432\r\n write(2 from 2) -> (20) \r\n - write(20 from 417) -> (0) lookup changegroupsu + write(20 from 432) -> (0) lookup changegroupsu write limit reached; closing socket $ rm -f error.log Server sends incomplete headers for batch request - $ hg serve --config badserver.closeaftersendbytes=695 -p $HGPORT -d --pid-file=hg.pid -E error.log + $ hg serve --config badserver.closeaftersendbytes=710 -p $HGPORT -d --pid-file=hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS TODO this output is horrible @@ -323,13 +323,13 @@ readline(-1) -> (2?) host: localhost:$HGPORT\r\n (glob) readline(-1) -> (49) user-agent: mercurial/proto-1.0 (Mercurial 4.2)\r\n readline(-1) -> (2) \r\n - write(36 from 36) -> (659) HTTP/1.1 200 Script output follows\r\n - write(23 from 23) -> (636) Server: badhttpserver\r\n - write(37 from 37) -> (599) Date: Fri, 14 Apr 2017 00:00:00 GMT\r\n - write(41 from 41) -> (558) Content-Type: application/mercurial-0.1\r\n - write(21 from 21) -> (537) Content-Length: 417\r\n - write(2 from 2) -> (535) \r\n - write(417 from 417) -> (118) lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 bundle2=HG20%0Abookmarks%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=none + write(36 from 36) -> (674) HTTP/1.1 200 Script output follows\r\n + write(23 from 23) -> (651) Server: badhttpserver\r\n + write(37 from 37) -> (614) Date: Fri, 14 Apr 2017 00:00:00 GMT\r\n + write(41 from 41) -> (573) Content-Type: application/mercurial-0.1\r\n + write(21 from 21) -> (552) Content-Length: 432\r\n + write(2 from 2) -> (550) \r\n + write(432 from 432) -> (118) lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_NO_PHASES_CAPS$ unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=none readline(65537) -> (26) GET /?cmd=batch HTTP/1.1\r\n readline(-1) -> (27) Accept-Encoding: identity\r\n readline(-1) -> (29) vary: X-HgArg-1,X-HgProto-1\r\n @@ -350,7 +350,7 @@ Server sends an incomplete HTTP response body to batch request - $ hg serve --config badserver.closeaftersendbytes=760 -p $HGPORT -d --pid-file=hg.pid -E error.log + $ hg serve --config badserver.closeaftersendbytes=775 -p $HGPORT -d --pid-file=hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS TODO client spews a stack due to uncaught ValueError in batch.results() @@ -371,13 +371,13 @@ readline(-1) -> (2?) host: localhost:$HGPORT\r\n (glob) readline(-1) -> (49) user-agent: mercurial/proto-1.0 (Mercurial 4.2)\r\n readline(-1) -> (2) \r\n - write(36 from 36) -> (724) HTTP/1.1 200 Script output follows\r\n - write(23 from 23) -> (701) Server: badhttpserver\r\n - write(37 from 37) -> (664) Date: Fri, 14 Apr 2017 00:00:00 GMT\r\n - write(41 from 41) -> (623) Content-Type: application/mercurial-0.1\r\n - write(21 from 21) -> (602) Content-Length: 417\r\n - write(2 from 2) -> (600) \r\n - write(417 from 417) -> (183) lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 bundle2=HG20%0Abookmarks%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=none + write(36 from 36) -> (739) HTTP/1.1 200 Script output follows\r\n + write(23 from 23) -> (716) Server: badhttpserver\r\n + write(37 from 37) -> (679) Date: Fri, 14 Apr 2017 00:00:00 GMT\r\n + write(41 from 41) -> (638) Content-Type: application/mercurial-0.1\r\n + write(21 from 21) -> (617) Content-Length: 432\r\n + write(2 from 2) -> (615) \r\n + write(432 from 432) -> (183) lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_NO_PHASES_CAPS$ unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=none readline(65537) -> (26) GET /?cmd=batch HTTP/1.1\r\n readline(-1) -> (27) Accept-Encoding: identity\r\n readline(-1) -> (29) vary: X-HgArg-1,X-HgProto-1\r\n @@ -400,7 +400,7 @@ Server sends incomplete headers for getbundle response - $ hg serve --config badserver.closeaftersendbytes=907 -p $HGPORT -d --pid-file=hg.pid -E error.log + $ hg serve --config badserver.closeaftersendbytes=922 -p $HGPORT -d --pid-file=hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS TODO this output is terrible @@ -423,13 +423,13 @@ readline(-1) -> (2?) host: localhost:$HGPORT\r\n (glob) readline(-1) -> (49) user-agent: mercurial/proto-1.0 (Mercurial 4.2)\r\n readline(-1) -> (2) \r\n - write(36 from 36) -> (871) HTTP/1.1 200 Script output follows\r\n - write(23 from 23) -> (848) Server: badhttpserver\r\n - write(37 from 37) -> (811) Date: Fri, 14 Apr 2017 00:00:00 GMT\r\n - write(41 from 41) -> (770) Content-Type: application/mercurial-0.1\r\n - write(21 from 21) -> (749) Content-Length: 417\r\n - write(2 from 2) -> (747) \r\n - write(417 from 417) -> (330) lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 bundle2=HG20%0Abookmarks%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=none + write(36 from 36) -> (886) HTTP/1.1 200 Script output follows\r\n + write(23 from 23) -> (863) Server: badhttpserver\r\n + write(37 from 37) -> (826) Date: Fri, 14 Apr 2017 00:00:00 GMT\r\n + write(41 from 41) -> (785) Content-Type: application/mercurial-0.1\r\n + write(21 from 21) -> (764) Content-Length: 432\r\n + write(2 from 2) -> (762) \r\n + write(432 from 432) -> (330) lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_NO_PHASES_CAPS$ unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=none readline(65537) -> (26) GET /?cmd=batch HTTP/1.1\r\n readline(-1) -> (27) Accept-Encoding: identity\r\n readline(-1) -> (29) vary: X-HgArg-1,X-HgProto-1\r\n @@ -449,7 +449,7 @@ readline(65537) -> (30) GET /?cmd=getbundle HTTP/1.1\r\n readline(-1) -> (27) Accept-Encoding: identity\r\n readline(-1) -> (29) vary: X-HgArg-1,X-HgProto-1\r\n - readline(-1) -> (422) x-hgarg-1: bookmarks=1&bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=1&common=0000000000000000000000000000000000000000&heads=96ee1d7354c4ad7372047672c36a1f561e3a6a4c&listkeys=phases%2Cbookmarks\r\n + readline(-1) -> (439) x-hgarg-1: bookmarks=1&bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Apartial-pull%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=1&common=0000000000000000000000000000000000000000&heads=96ee1d7354c4ad7372047672c36a1f561e3a6a4c&listkeys=phases%2Cbookmarks\r\n readline(-1) -> (48) x-hgproto-1: 0.1 0.2 comp=$USUAL_COMPRESSIONS$\r\n readline(-1) -> (35) accept: application/mercurial-0.1\r\n readline(-1) -> (2?) host: localhost:$HGPORT\r\n (glob) @@ -466,7 +466,7 @@ Server sends empty HTTP body for getbundle - $ hg serve --config badserver.closeaftersendbytes=945 -p $HGPORT -d --pid-file=hg.pid -E error.log + $ hg serve --config badserver.closeaftersendbytes=960 -p $HGPORT -d --pid-file=hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS $ hg clone http://localhost:$HGPORT/ clone @@ -484,13 +484,13 @@ readline(-1) -> (2?) host: localhost:$HGPORT\r\n (glob) readline(-1) -> (49) user-agent: mercurial/proto-1.0 (Mercurial 4.2)\r\n readline(-1) -> (2) \r\n - write(36 from 36) -> (909) HTTP/1.1 200 Script output follows\r\n - write(23 from 23) -> (886) Server: badhttpserver\r\n - write(37 from 37) -> (849) Date: Fri, 14 Apr 2017 00:00:00 GMT\r\n - write(41 from 41) -> (808) Content-Type: application/mercurial-0.1\r\n - write(21 from 21) -> (787) Content-Length: 417\r\n - write(2 from 2) -> (785) \r\n - write(417 from 417) -> (368) lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 bundle2=HG20%0Abookmarks%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=none + write(36 from 36) -> (924) HTTP/1.1 200 Script output follows\r\n + write(23 from 23) -> (901) Server: badhttpserver\r\n + write(37 from 37) -> (864) Date: Fri, 14 Apr 2017 00:00:00 GMT\r\n + write(41 from 41) -> (823) Content-Type: application/mercurial-0.1\r\n + write(21 from 21) -> (802) Content-Length: 432\r\n + write(2 from 2) -> (800) \r\n + write(432 from 432) -> (368) lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_NO_PHASES_CAPS$ unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=none readline(65537) -> (26) GET /?cmd=batch HTTP/1.1\r\n readline(-1) -> (27) Accept-Encoding: identity\r\n readline(-1) -> (29) vary: X-HgArg-1,X-HgProto-1\r\n @@ -510,7 +510,7 @@ readline(65537) -> (30) GET /?cmd=getbundle HTTP/1.1\r\n readline(-1) -> (27) Accept-Encoding: identity\r\n readline(-1) -> (29) vary: X-HgArg-1,X-HgProto-1\r\n - readline(-1) -> (422) x-hgarg-1: bookmarks=1&bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=1&common=0000000000000000000000000000000000000000&heads=96ee1d7354c4ad7372047672c36a1f561e3a6a4c&listkeys=phases%2Cbookmarks\r\n + readline(-1) -> (439) x-hgarg-1: bookmarks=1&bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Apartial-pull%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=1&common=0000000000000000000000000000000000000000&heads=96ee1d7354c4ad7372047672c36a1f561e3a6a4c&listkeys=phases%2Cbookmarks\r\n readline(-1) -> (48) x-hgproto-1: 0.1 0.2 comp=$USUAL_COMPRESSIONS$\r\n readline(-1) -> (35) accept: application/mercurial-0.1\r\n readline(-1) -> (2?) host: localhost:$HGPORT\r\n (glob) @@ -529,7 +529,7 @@ Server sends partial compression string - $ hg serve --config badserver.closeaftersendbytes=969 -p $HGPORT -d --pid-file=hg.pid -E error.log + $ hg serve --config badserver.closeaftersendbytes=984 -p $HGPORT -d --pid-file=hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS $ hg clone http://localhost:$HGPORT/ clone @@ -547,13 +547,13 @@ readline(-1) -> (2?) host: localhost:$HGPORT\r\n (glob) readline(-1) -> (49) user-agent: mercurial/proto-1.0 (Mercurial 4.2)\r\n readline(-1) -> (2) \r\n - write(36 from 36) -> (933) HTTP/1.1 200 Script output follows\r\n - write(23 from 23) -> (910) Server: badhttpserver\r\n - write(37 from 37) -> (873) Date: Fri, 14 Apr 2017 00:00:00 GMT\r\n - write(41 from 41) -> (832) Content-Type: application/mercurial-0.1\r\n - write(21 from 21) -> (811) Content-Length: 417\r\n - write(2 from 2) -> (809) \r\n - write(417 from 417) -> (392) lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 bundle2=HG20%0Abookmarks%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=none + write(36 from 36) -> (948) HTTP/1.1 200 Script output follows\r\n + write(23 from 23) -> (925) Server: badhttpserver\r\n + write(37 from 37) -> (888) Date: Fri, 14 Apr 2017 00:00:00 GMT\r\n + write(41 from 41) -> (847) Content-Type: application/mercurial-0.1\r\n + write(21 from 21) -> (826) Content-Length: 432\r\n + write(2 from 2) -> (824) \r\n + write(432 from 432) -> (392) lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_NO_PHASES_CAPS$ unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=none readline(65537) -> (26) GET /?cmd=batch HTTP/1.1\r\n readline(-1) -> (27) Accept-Encoding: identity\r\n readline(-1) -> (29) vary: X-HgArg-1,X-HgProto-1\r\n @@ -573,7 +573,7 @@ readline(65537) -> (30) GET /?cmd=getbundle HTTP/1.1\r\n readline(-1) -> (27) Accept-Encoding: identity\r\n readline(-1) -> (29) vary: X-HgArg-1,X-HgProto-1\r\n - readline(-1) -> (422) x-hgarg-1: bookmarks=1&bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=1&common=0000000000000000000000000000000000000000&heads=96ee1d7354c4ad7372047672c36a1f561e3a6a4c&listkeys=phases%2Cbookmarks\r\n + readline(-1) -> (439) x-hgarg-1: bookmarks=1&bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Apartial-pull%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=1&common=0000000000000000000000000000000000000000&heads=96ee1d7354c4ad7372047672c36a1f561e3a6a4c&listkeys=phases%2Cbookmarks\r\n readline(-1) -> (48) x-hgproto-1: 0.1 0.2 comp=$USUAL_COMPRESSIONS$\r\n readline(-1) -> (35) accept: application/mercurial-0.1\r\n readline(-1) -> (2?) host: localhost:$HGPORT\r\n (glob) @@ -595,7 +595,7 @@ Server sends partial bundle2 header magic - $ hg serve --config badserver.closeaftersendbytes=966 -p $HGPORT -d --pid-file=hg.pid -E error.log + $ hg serve --config badserver.closeaftersendbytes=981 -p $HGPORT -d --pid-file=hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS $ hg clone http://localhost:$HGPORT/ clone @@ -619,7 +619,7 @@ Server sends incomplete bundle2 stream params length - $ hg serve --config badserver.closeaftersendbytes=975 -p $HGPORT -d --pid-file=hg.pid -E error.log + $ hg serve --config badserver.closeaftersendbytes=990 -p $HGPORT -d --pid-file=hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS $ hg clone http://localhost:$HGPORT/ clone @@ -644,7 +644,7 @@ Servers stops after bundle2 stream params header - $ hg serve --config badserver.closeaftersendbytes=978 -p $HGPORT -d --pid-file=hg.pid -E error.log + $ hg serve --config badserver.closeaftersendbytes=993 -p $HGPORT -d --pid-file=hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS $ hg clone http://localhost:$HGPORT/ clone @@ -669,7 +669,7 @@ Server stops sending after bundle2 part header length - $ hg serve --config badserver.closeaftersendbytes=987 -p $HGPORT -d --pid-file=hg.pid -E error.log + $ hg serve --config badserver.closeaftersendbytes=1002 -p $HGPORT -d --pid-file=hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS $ hg clone http://localhost:$HGPORT/ clone @@ -695,7 +695,7 @@ Server stops sending after bundle2 part header - $ hg serve --config badserver.closeaftersendbytes=1034 -p $HGPORT -d --pid-file=hg.pid -E error.log + $ hg serve --config badserver.closeaftersendbytes=1049 -p $HGPORT -d --pid-file=hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS $ hg clone http://localhost:$HGPORT/ clone @@ -725,7 +725,7 @@ Server stops after bundle2 part payload chunk size - $ hg serve --config badserver.closeaftersendbytes=1055 -p $HGPORT -d --pid-file=hg.pid -E error.log + $ hg serve --config badserver.closeaftersendbytes=1070 -p $HGPORT -d --pid-file=hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS $ hg clone http://localhost:$HGPORT/ clone @@ -756,7 +756,7 @@ Server stops sending in middle of bundle2 payload chunk - $ hg serve --config badserver.closeaftersendbytes=1516 -p $HGPORT -d --pid-file=hg.pid -E error.log + $ hg serve --config badserver.closeaftersendbytes=1531 -p $HGPORT -d --pid-file=hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS $ hg clone http://localhost:$HGPORT/ clone @@ -788,7 +788,7 @@ Server stops sending after 0 length payload chunk size - $ hg serve --config badserver.closeaftersendbytes=1547 -p $HGPORT -d --pid-file=hg.pid -E error.log + $ hg serve --config badserver.closeaftersendbytes=1562 -p $HGPORT -d --pid-file=hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS $ hg clone http://localhost:$HGPORT/ clone @@ -825,7 +825,7 @@ Server stops sending after 0 part bundle part header (indicating end of bundle2 payload) This is before the 0 size chunked transfer part that signals end of HTTP response. - $ hg serve --config badserver.closeaftersendbytes=1722 -p $HGPORT -d --pid-file=hg.pid -E error.log + $ hg serve --config badserver.closeaftersendbytes=1737 -p $HGPORT -d --pid-file=hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS $ hg clone http://localhost:$HGPORT/ clone @@ -869,7 +869,7 @@ Server sends a size 0 chunked-transfer size without terminating \r\n - $ hg serve --config badserver.closeaftersendbytes=1725 -p $HGPORT -d --pid-file=hg.pid -E error.log + $ hg serve --config badserver.closeaftersendbytes=1740 -p $HGPORT -d --pid-file=hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS $ hg clone http://localhost:$HGPORT/ clone diff --git a/tests/test-help.t b/tests/test-help.t --- a/tests/test-help.t +++ b/tests/test-help.t @@ -1487,6 +1487,7 @@ bookmarks create a new bookmark or list existing bookmarks clone make a copy of an existing repository paths show aliases for remote repositories + pull pull changes from the specified source update update working directory (or switch revisions) Extensions: diff --git a/tests/test-debugcommands.t b/tests/test-debugcommands.t --- a/tests/test-debugcommands.t +++ b/tests/test-debugcommands.t @@ -373,6 +373,7 @@ pushkey hgtagsfnodes listkeys + partial-pull phases heads pushkey diff --git a/tests/test-bookmarks-pushpull.t b/tests/test-bookmarks-pushpull.t --- a/tests/test-bookmarks-pushpull.t +++ b/tests/test-bookmarks-pushpull.t @@ -129,10 +129,10 @@ bundle2-output: bundle parameter: bundle2-output: start of parts bundle2-output: bundle part: "replycaps" - bundle2-output-part: "replycaps" 195 bytes payload + bundle2-output-part: "replycaps" 208 bytes payload bundle2-output: part 0: "REPLYCAPS" bundle2-output: header chunk size: 16 - bundle2-output: payload chunk size: 195 + bundle2-output: payload chunk size: 208 bundle2-output: closing payload chunk bundle2-output: bundle part: "check:bookmarks" bundle2-output-part: "check:bookmarks" 23 bytes payload @@ -162,9 +162,9 @@ bundle2-input: part parameters: 0 bundle2-input: found a handler for part replycaps bundle2-input-part: "replycaps" supported - bundle2-input: payload chunk size: 195 + bundle2-input: payload chunk size: 208 bundle2-input: payload chunk size: 0 - bundle2-input-part: total payload size 195 + bundle2-input-part: total payload size 208 bundle2-input: part header size: 22 bundle2-input: part type: "CHECK:BOOKMARKS" bundle2-input: part id: "1" @@ -241,10 +241,10 @@ bundle2-output: bundle parameter: bundle2-output: start of parts bundle2-output: bundle part: "replycaps" - bundle2-output-part: "replycaps" 195 bytes payload + bundle2-output-part: "replycaps" 208 bytes payload bundle2-output: part 0: "REPLYCAPS" bundle2-output: header chunk size: 16 - bundle2-output: payload chunk size: 195 + bundle2-output: payload chunk size: 208 bundle2-output: closing payload chunk bundle2-output: bundle part: "check:bookmarks" bundle2-output-part: "check:bookmarks" 23 bytes payload @@ -275,9 +275,9 @@ bundle2-input: part parameters: 0 bundle2-input: found a handler for part replycaps bundle2-input-part: "replycaps" supported - bundle2-input: payload chunk size: 195 + bundle2-input: payload chunk size: 208 bundle2-input: payload chunk size: 0 - bundle2-input-part: total payload size 195 + bundle2-input-part: total payload size 208 bundle2-input: part header size: 22 bundle2-input: part type: "CHECK:BOOKMARKS" bundle2-input: part id: "1" diff --git a/tests/test-acl.t b/tests/test-acl.t --- a/tests/test-acl.t +++ b/tests/test-acl.t @@ -93,14 +93,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -156,14 +156,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -222,14 +222,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -298,14 +298,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -366,14 +366,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -439,14 +439,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -509,14 +509,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -584,14 +584,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -656,14 +656,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -730,14 +730,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -813,14 +813,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -894,14 +894,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -970,14 +970,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -1057,14 +1057,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -1143,14 +1143,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -1225,14 +1225,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -1304,14 +1304,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -1387,14 +1387,14 @@ f9cafe1212c8c6fa1120d14a556e18cc44ff8bdd 911600dab2ae7a9baff75958b84fe606851ce955 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 24 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 24 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 24 bundle2-input-part: "check:heads" supported @@ -1507,14 +1507,14 @@ 911600dab2ae7a9baff75958b84fe606851ce955 e8fc755d4d8217ee5b0c2bb41558c40d43b92c01 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 48 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 48 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 48 bundle2-input-part: "check:heads" supported @@ -1591,14 +1591,14 @@ 911600dab2ae7a9baff75958b84fe606851ce955 e8fc755d4d8217ee5b0c2bb41558c40d43b92c01 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 48 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 48 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 48 bundle2-input-part: "check:heads" supported @@ -1668,14 +1668,14 @@ 911600dab2ae7a9baff75958b84fe606851ce955 e8fc755d4d8217ee5b0c2bb41558c40d43b92c01 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 48 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 48 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 48 bundle2-input-part: "check:heads" supported @@ -1741,14 +1741,14 @@ 911600dab2ae7a9baff75958b84fe606851ce955 e8fc755d4d8217ee5b0c2bb41558c40d43b92c01 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 48 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 48 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 48 bundle2-input-part: "check:heads" supported @@ -1808,14 +1808,14 @@ 911600dab2ae7a9baff75958b84fe606851ce955 e8fc755d4d8217ee5b0c2bb41558c40d43b92c01 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 48 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 48 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 48 bundle2-input-part: "check:heads" supported @@ -1897,14 +1897,14 @@ 911600dab2ae7a9baff75958b84fe606851ce955 e8fc755d4d8217ee5b0c2bb41558c40d43b92c01 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 48 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 48 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 48 bundle2-input-part: "check:heads" supported @@ -1985,14 +1985,14 @@ 911600dab2ae7a9baff75958b84fe606851ce955 e8fc755d4d8217ee5b0c2bb41558c40d43b92c01 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 48 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 48 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 48 bundle2-input-part: "check:heads" supported @@ -2057,14 +2057,14 @@ 911600dab2ae7a9baff75958b84fe606851ce955 e8fc755d4d8217ee5b0c2bb41558c40d43b92c01 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 48 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 48 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 48 bundle2-input-part: "check:heads" supported @@ -2139,14 +2139,14 @@ 911600dab2ae7a9baff75958b84fe606851ce955 e8fc755d4d8217ee5b0c2bb41558c40d43b92c01 bundle2-output-bundle: "HG20", 5 parts total - bundle2-output-part: "replycaps" 178 bytes payload + bundle2-output-part: "replycaps" 191 bytes payload bundle2-output-part: "check:phases" 48 bytes payload bundle2-output-part: "check:heads" streamed payload bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload bundle2-output-part: "phase-heads" 48 bytes payload bundle2-input-bundle: with-transaction bundle2-input-part: "replycaps" supported - bundle2-input-part: total payload size 178 + bundle2-input-part: total payload size 191 bundle2-input-part: "check:phases" supported bundle2-input-part: total payload size 48 bundle2-input-part: "check:heads" supported diff --git a/tests/common-pattern.py b/tests/common-pattern.py --- a/tests/common-pattern.py +++ b/tests/common-pattern.py @@ -16,6 +16,7 @@ br'error%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250A' br'hgtagsfnodes%250A' br'listkeys%250A' + br'partial-pull%250A' br'phases%253Dheads%250A' br'pushkey%250A' br'remote-changegroup%253Dhttp%252Chttps', @@ -30,12 +31,26 @@ br'error%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0A' br'hgtagsfnodes%0A' br'listkeys%0A' + br'partial-pull%0A' br'phases%3Dheads%0A' br'pushkey%0A' br'remote-changegroup%3Dhttp%2Chttps', # (replacement patterns) br'$USUAL_BUNDLE2_CAPS$' ), + (br'bundle2=HG20%0A' + br'bookmarks%0A' + br'changegroup%3D01%2C02%0A' + br'digests%3Dmd5%2Csha1%2Csha512%0A' + br'error%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0A' + br'hgtagsfnodes%0A' + br'listkeys%0A' + br'partial-pull%0A' + br'pushkey%0A' + br'remote-changegroup%3Dhttp%2Chttps', + # (replacement patterns) + br'$USUAL_BUNDLE2_NO_PHASES_CAPS$' + ), # HTTP log dates (br' - - \[\d\d/.../2\d\d\d \d\d:\d\d:\d\d] "GET', br' - - [$LOGDATE$] "GET' diff --git a/mercurial/wireproto.py b/mercurial/wireproto.py --- a/mercurial/wireproto.py +++ b/mercurial/wireproto.py @@ -831,6 +831,65 @@ opts = options('debugwireargs', ['three', 'four'], others) return repo.debugwireargs(one, two, **pycompat.strkwargs(opts)) +def find_pullbundle(repo, opts, clheads, heads, common): + """Return a file object for the first matching pullbundle. + + Pullbundles are specified in .hg/pullbundles.manifest similar to + clonebundles. + For each entry, the bundle specification is checked for compatibility: + - Client features vs the BUNDLESPEC. + - Revisions shared with the clients vs base revisions of the bundle. + A bundle can be applied only if all its base revisions are known by + the client. + - At least one leaf of the bundle's DAG is missing on the client. + - Every leaf of the bundle's DAG is part of node set the client wants. + E.g. do not send a bundle of all changes if the client wants only + one specific branch of many. + """ + def decodehexstring(s): + return set([h.decode('hex') for h in s.split(';')]) + + manifest = repo.vfs.tryread('pullbundles.manifest') + if not manifest: + return None + res = exchange.parseclonebundlesmanifest(repo, manifest) + res = exchange.filterclonebundleentries(repo, res) + if not res: + return None + cl = repo.changelog + heads_anc = cl.ancestors([cl.rev(rev) for rev in heads], inclusive=True) + common_anc = cl.ancestors([cl.rev(rev) for rev in common], inclusive=True) + for entry in res: + if 'heads' in entry: + try: + bundle_heads = decodehexstring(entry['heads']) + except TypeError: + # Bad heads entry + continue + if bundle_heads.issubset(common): + continue # Nothing new + if all(cl.rev(rev) in common_anc for rev in bundle_heads): + continue # Still nothing new + if any(cl.rev(rev) not in heads_anc and + cl.rev(rev) not in common_anc for rev in bundle_heads): + continue + if 'bases' in entry: + try: + bundle_bases = decodehexstring(entry['bases']) + except TypeError: + # Bad bases entry + continue + if not all(cl.rev(rev) in common_anc for rev in bundle_bases): + continue + path = entry['URL'] + repo.ui.debug('sending pullbundle "%s"\n' % path) + try: + return repo.vfs.open(path) + except IOError: + repo.ui.debug('pullbundle "%s" not accessible\n' % path) + continue + return None + @wireprotocommand('getbundle', '*') def getbundle(repo, proto, others): opts = options('getbundle', gboptsmap.keys(), others) @@ -861,12 +920,26 @@ hint=bundle2requiredhint) try: + clheads = set(repo.changelog.heads()) + heads = set(opts.get('heads', set())) + common = set(opts.get('common', set())) + common.discard(nullid) + b2caps = {} + for bcaps in opts.get('bundlecaps', []): + if bcaps.startswith('bundle2='): + blob = urlreq.unquote(bcaps[len('bundle2='):]) + b2caps.update(bundle2.decodecaps(blob)) + + if (repo.ui.configbool('server', 'pullbundle') and + 'partial-pull' in b2caps): + # Check if a pre-built bundle covers this request. + bundle = find_pullbundle(repo, opts, clheads, heads, common) + if bundle: + return streamres(gen=util.filechunkiter(bundle), + prefer_uncompressed=True) + if repo.ui.configbool('server', 'disablefullbundle'): # Check to see if this is a full clone. - clheads = set(repo.changelog.heads()) - heads = set(opts.get('heads', set())) - common = set(opts.get('common', set())) - common.discard(nullid) if not common and clheads == heads: raise error.Abort( _('server has pull-based clones disabled'), diff --git a/mercurial/help/config.txt b/mercurial/help/config.txt --- a/mercurial/help/config.txt +++ b/mercurial/help/config.txt @@ -1772,6 +1772,14 @@ are highly recommended. Partial clones will still be allowed. (default: False) +``pullbundle`` + When set, the server will check pullbundle.manifest for bundles + covering the requested heads and common nodes. The first matching + entry will be streamed to the client. + + For HTTP transport, the stream will still use zlib compression + for older clients. + ``concurrent-push-mode`` Level of allowed race condition between two pushing clients. diff --git a/mercurial/exchange.py b/mercurial/exchange.py --- a/mercurial/exchange.py +++ b/mercurial/exchange.py @@ -1353,7 +1353,36 @@ streamclone.maybeperformlegacystreamclone(pullop) _pulldiscovery(pullop) if pullop.canusebundle2: - _pullbundle2(pullop) + # The server may send a partial reply, i.e. when inlining + # pre-computed bundles. In that case, update the common + # set based on the results and pull another bundle. + # + # There are two indicators that the process is finished: + # - no changeset has been added, or + # - all remote heads are known locally. + # The head check must use the unfiltered view as obsoletion + # markers can hide heads. + unfi = repo.unfiltered() + unficl = unfi.changelog + def headsofdiff(h1, h2): + """Returns heads(h1 % h2)""" + res = unfi.set('heads(%ln %% %ln)', h1, h2) + return set(ctx.node() for ctx in res) + def headsofunion(h1, h2): + """Returns heads((h1 + h2) - null)""" + res = unfi.set('heads((%ln + %ln - null))', h1, h2) + return set(ctx.node() for ctx in res) + while True: + old_heads = unficl.heads() + clstart = len(unficl) + _pullbundle2(pullop) + if clstart == len(unficl): + break + if all(unficl.hasnode(n) for n in pullop.rheads): + break + new_heads = headsofdiff(unficl.heads(), old_heads) + pullop.common = headsofunion(new_heads, pullop.common) + pullop.rheads = set(pullop.rheads) - pullop.common _pullchangeset(pullop) _pullphase(pullop) _pullbookmarks(pullop) diff --git a/mercurial/configitems.py b/mercurial/configitems.py --- a/mercurial/configitems.py +++ b/mercurial/configitems.py @@ -886,6 +886,9 @@ coreconfigitem('server', 'disablefullbundle', default=False, ) +coreconfigitem('server', 'pullbundle', + default=False, +) coreconfigitem('server', 'maxhttpheaderlen', default=1024, ) diff --git a/mercurial/commands.py b/mercurial/commands.py --- a/mercurial/commands.py +++ b/mercurial/commands.py @@ -1389,12 +1389,13 @@ i) tip When cloning from servers that support it, Mercurial may fetch - pre-generated data from a server-advertised URL. When this is done, - hooks operating on incoming changesets and changegroups may fire twice, - once for the bundle fetched from the URL and another for any additional - data not fetched from this URL. In addition, if an error occurs, the - repository may be rolled back to a partial clone. This behavior may - change in future releases. See :hg:`help -e clonebundles` for more. + pre-generated data from a server-advertised URL or inline from the + same stream. When this is done, hooks operating on incoming changesets + and changegroups may fire more than once, once for each pre-generated + bundle and as well as for any additional remaining data. In addition, + if an error occurs, the repository may be rolled back to a partial + clone. This behavior may change in future releases. + See :hg:`help -e clonebundles` for more. Examples: @@ -3953,6 +3954,12 @@ -R is specified). By default, this does not update the copy of the project in the working directory. + When cloning from servers that support it, Mercurial may fetch + pre-generated data. When this is done, hooks operating on incoming + changesets and changegroups may fire more than once, once for each + pre-generated bundle and as well as for any additional remaining + data. See :hg:`help -e clonebundles` for more. + Use :hg:`incoming` if you want to see what would have been added by a pull at the time you issued this command. If you then decide to add those changes to the repository, you should use :hg:`pull diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py --- a/mercurial/bundle2.py +++ b/mercurial/bundle2.py @@ -1486,6 +1486,7 @@ 'remote-changegroup': ('http', 'https'), 'hgtagsfnodes': (), 'phases': ('heads',), + 'partial-pull': (), } def getrepocaps(repo, allowpushback=False): diff --git a/hgext/clonebundles.py b/hgext/clonebundles.py --- a/hgext/clonebundles.py +++ b/hgext/clonebundles.py @@ -6,7 +6,8 @@ "clonebundles" is a server-side extension used to advertise the existence of pre-generated, externally hosted bundle files to clients that are cloning so that cloning can be faster, more reliable, and require less -resources on the server. +resources on the server. "pullbundles" is a related feature for sending +pre-generated bundle files to clients as part of pull operations. Cloning can be a CPU and I/O intensive operation on servers. Traditionally, the server, in response to a client's request to clone, dynamically generates @@ -16,8 +17,12 @@ servers with large repositories or with high clone volume, the load from clones can make scaling the server challenging and costly. -This extension provides server operators the ability to offload potentially -expensive clone load to an external service. Here's how it works. +This extension provides server operators the ability to offload +potentially expensive clone load to an external service. Pre-generated +bundles also allow using more CPU intensive compression, reducing the +effective bandwidth requirements. + +Here's how clone bundles work: 1. A server operator establishes a mechanism for making bundle files available on a hosting service where Mercurial clients can fetch them. @@ -33,22 +38,36 @@ 7. The client reconnects to the original server and performs the equivalent of :hg:`pull` to retrieve all repository data not in the bundle. (The repository could have been updated between when the bundle was created - and when the client started the clone.) + and when the client started the clone.) This may use "pullbundles". Instead of the server generating full repository bundles for every clone request, it generates full bundles once and they are subsequently reused to bootstrap new clones. The server may still transfer data at clone time. However, this is only data that has been added/changed since the bundle was created. For large, established repositories, this can reduce server load for clones to less than 1% of original. +Here's how pullbundles work: + +1. A manifest file listing available bundles and describing the revisions + is added to the Mercurial repository on the server. +2. A new-enough client informs the server that it supports partial pulls + and initiates a pull. +3. If the server has pull bundles enabled and sees the client advertising + partial pulls, it checks for a matching pull bundle in the manifest. + A bundle matches if the format is supported by the client, the client + has the required revisions already and needs something from the bundle. +4. If there is at least one matching bundle, the server sends it to the client. +5. The client applies the bundle and notices that the server reply was + incomplete. It initiates another pull. + To work, this extension requires the following of server operators: * Generating bundle files of repository content (typically periodically, such as once per day). -* A file server that clients have network access to and that Python knows - how to talk to through its normal URL handling facility (typically an - HTTP server). +* Clone bundles: A file server that clients have network access to and that + Python knows how to talk to through its normal URL handling facility + (typically an HTTP/HTTPS server). * A process for keeping the bundles manifest in sync with available bundle files. @@ -61,7 +80,7 @@ :hg:`bundle --all` is used to produce a bundle of the entire repository. :hg:`debugcreatestreamclonebundle` can be used to produce a special -*streaming clone bundle*. These are bundle files that are extremely efficient +*streaming clonebundle*. These are bundle files that are extremely efficient to produce and consume (read: fast). However, they are larger than traditional bundle formats and require that clients support the exact set of repository data store formats in use by the repository that created them. @@ -73,7 +92,8 @@ A server operator is responsible for creating a ``.hg/clonebundles.manifest`` file containing the list of available bundle files suitable for seeding clones. If this file does not exist, the repository will not advertise the -existence of clone bundles when clients connect. +existence of clone bundles when clients connect. For pull bundles, +``.hg/pullbundles.manifest`` is used. The manifest file contains a newline (\\n) delimited list of entries. @@ -85,6 +105,9 @@ pairs describing additional properties of this bundle. Both keys and values are URI encoded. +For pull bundles, the URL is a path under the ``.hg`` directory of the +repository. + Keys in UPPERCASE are reserved for use by Mercurial and are defined below. All non-uppercase keys can be used by site installations. An example use for custom properties is to use the *datacenter* attribute to define which @@ -133,6 +156,15 @@ Value should be "true". +heads + Used for pull bundles. This contains the ``;`` separated changeset + hashes of the heads of the bundle content. + +bases + Used for pull bundles. This contains the ``;`` separated changeset + hashes of the roots of the bundle content. This can be skipped if + the bundle was created without ``--base``. + Manifests can contain multiple entries. Assuming metadata is defined, clients will filter entries from the manifest that they don't support. The remaining entries are optionally sorted by client preferences To: joerg.sonnenberger, #hg-reviewers, indygreg Cc: indygreg, durin42, mercurial-devel _______________________________________________ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
