On 10/12/2017 05:52 PM, Ian Wienand wrote: > I tried this in order, firstly recreating references.db (didn't help) > and so I have started the checksums.db recreation. This is now > running; I just moved the old one out of the way
Well, that didn't go so well. The output flooded stuff and then it died. --- ... Within references.db subtable references at get: No such file or directory BDB0134 read: 0x11989b0, 4096: No such file or directory Internal error of the underlying BerkeleyDB database: Within references.db subtable references at get: No such file or directory BDB0134 read: 0x11989b0, 4096: No such file or directory Internal error of the underlying BerkeleyDB database: Within references.db subtable references at get: No such file or directory 37 files were added but not used. The next deleteunreferenced call will delete them. BDB0151 fsync: Connection timed out BDB0164 close: Connection timed out ./db/checksums.db: Connection timed out BDB3028 ./db/checksums.db: unable to flush: Connection timed out db_close(checksums.db, pool): Connection timed out Error creating './db/version.new': Connection timed out(errno is 110) Error 110 deleting lock file './db/lockfile': Connection timed out! There have been errors! --- Presumably this matches up with the AFS errors logged --- [Thu Oct 12 09:19:59 2017] afs: Lost contact with file server 104.130.138.161 in cell openstack.org (code -512) (all multi-homed ip addresses down for the server) [Thu Oct 12 09:19:59 2017] afs: Lost contact with file server 104.130.138.161 in cell openstack.org (code -512) (all multi-homed ip addresses down for the server) [Thu Oct 12 09:19:59 2017] afs: failed to store file (110) [Thu Oct 12 09:20:02 2017] afs: failed to store file (110) [Thu Oct 12 09:20:10 2017] afs: file server 104.130.138.161 in cell openstack.org is back up (code 0) (multi-homed address; other same-host interfaces may still be down) [Thu Oct 12 09:20:10 2017] afs: file server 104.130.138.161 in cell openstack.org is back up (code 0) (multi-homed address; other same-host interfaces may still be down) --- I restarted for good luck, but if this is transient network issues, I guess it will just happen again. ping shows no packet loss, but very occasional latency spikes, fwiw. We restarted mirror-update; maybe it's worth restarting the AFS servers too? -i _______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
