Faidon has uploaded a new change for review.
https://gerrit.wikimedia.org/r/63370
Change subject: swiftrepl: fetch all containers, not just the first 10k
......................................................................
swiftrepl: fetch all containers, not just the first 10k
Doh!
Change-Id: I464b6a68e420551ec2dea5056669290df8f2360c
---
M swiftrepl/swiftrepl.py
1 file changed, 14 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/software
refs/changes/70/63370/1
diff --git a/swiftrepl/swiftrepl.py b/swiftrepl/swiftrepl.py
index 39ef2d0..c5a646f 100644
--- a/swiftrepl/swiftrepl.py
+++ b/swiftrepl/swiftrepl.py
@@ -492,7 +492,20 @@
dstconnpool = connect(dst)
srcconn = srcconnpool.get()
- containerlist = [container for container in srcconn.get_all_containers()
+
+ containers=[]
+ limit=10000
+ last=''
+ while True:
+ page = srcconn.get_all_containers(limit=limit, marker=last)
+ if len(page) == 0:
+ break
+ last = page[-1].name.encode("utf-8")
+ containers.extend(page)
+ if len(page) < limit:
+ break
+
+ containerlist = [container for container in containers
if re.match(container_regexp,
container.name)]
if '-r' in sys.argv: random.shuffle(containerlist)
containers = collections.deque(containerlist)
--
To view, visit https://gerrit.wikimedia.org/r/63370
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I464b6a68e420551ec2dea5056669290df8f2360c
Gerrit-PatchSet: 1
Gerrit-Project: operations/software
Gerrit-Branch: master
Gerrit-Owner: Faidon <fai...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
