(moving to infra) On 10/12/2017 04:28 PM, Ian Wienand wrote: > mirrors provide, leading apt to great confusion. Some debugging notes > on reprepro at [1], but I have to conclude the .db files are corrupt > and I have no idea how to recreate these other than to start again.
I ran the reprepro under strace, and the last thing that comes out is 3170 pread(6, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 90521600) = 4096 then it just stops with cpu at 100%. lsof tells us reprepro 3170 root 6u REG 0,25 90628096 2537568 /afs/.openstack.org/mirror/ubuntu/db/checksums.db so, that db seems as likely as any to be causing the problems pabelanger pointed to some recovery instructions at [1] previously. I tried this in order, firstly recreating references.db (didn't help) and so I have started the checksums.db recreation. This is now running; I just moved the old one out of the way root@mirror-update:/afs/.openstack.org/mirror/ubuntu/db# ls -lh total 1.1G -rw-r--r-- 1 10004 root 1.6M Oct 12 06:38 checksums.db -rw-r--r-- 1 10004 root 87M Oct 12 02:59 checksums.db.old This started at about 06:30, meaning ~5 minutes/mb so I think around 6 hours till this is finished, hopefully (it's dragging everything across afs). Please take any of this over; it's running on mirror-update: screen(9683)─┬─bash(9684)───su(9917)───bash(9918) │ ├─bash(10466)───k5start(3755)───bash(3758)─┬─find(3996) │ └─reprepro(3997) note i'm holding the cron lock with root 10957 9918 0 06:46 pts/0 00:00:00 flock -n /var/run/reprepro/ubuntu.lock bash -c while true; do sleep 1000; done (ps, I think we need to make those dirs on reboot: https://review.openstack.org/511380) -i [1] https://github.com/esc/reprepro/blob/master/docs/recovery _______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra