> Il giorno 29 ago 2017, alle ore 14:14, Jean-Louis Martineau > <jmartin...@carbonite.com> ha scritto: > > Are you using the same bucket for another amanda configuration?
As far as I can see, no. But then again, I could very well be missing something obvious /etc/amanda/amanda.conf: > infofile "/var/lib/amanda/state/curinfo" > indexdir "/var/lib/amanda/state/index" > dumpuser "backup" > mailto “<xxx>" > > define changer s3 { > tapedev > "chg-multi:s3:onebip-backups-admin/slot-bb2e601e63b3408bb6a865e442a28366-{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}" > changerfile "/var/lib/amanda/state/s3-statefile" > device-property "NB_THREADS_BACKUP" "8" > device-property "NB_THREADS_RECOVERY" "8" > device_property "S3_ACCESS_KEY” “<>" > device_property "S3_SECRET_KEY” “<>" > device_property "S3_BUCKET_LOCATION" "eu-west-1" > device_property "S3_SSL" "YES" > device_property "BLOCK_SIZE" "10 megabytes" > } > > tpchanger "s3" > tapetype S3 > > define tapetype S3 { > comment "S3 Bucket" > length 10240 gigabytes > } /etc/amanda/zfs-dataset/amanda.conf: > includefile "../amanda.conf” > > org "zfs-dataset" > logdir "/var/log/zfs-dataset" > > define application-tool zfs-dataset-app { > comment "amzfs-sendrecv" > plugin "amzfs-sendrecv" > property "ZFS-PATH" "/sbin/zfs" > property "PFEXEC-PATH" "/usr/bin/sudo" > property "PFEXEC" "YES" > } > > define dumptype zfs-dataset-dump { > program "APPLICATION" > application "zfs-dataset-app" > auth "ssh" > ssh_keys "/var/backups/.ssh/id_rsa" > } /etc/amanda/zfs-dataset/zfs-dataset/disklist: > ci-master rpool/jenkins zfs-dataset-dump > backup-server rpool/amanda-conf zfs-dataset-dump > backup-server rpool/amanda-home zfs-dataset-dump > backup-server rpool/amanda-state zfs-dataset-dump > vpn-server rpool/openvpn zfs-dataset-dump root@backup-server:/etc/amanda# cat zfs-dataset/tapelist: > 20170829000017 slot-13 reuse BLOCKSIZE:10240 > 20170828000017 slot-12 reuse BLOCKSIZE:10240 > 20170827000017 slot-11 reuse BLOCKSIZE:10240 > 20170826000017 slot-10 reuse BLOCKSIZE:10240 > 20170825000017 slot-9 reuse BLOCKSIZE:10240 > 20170824000017 slot-8 reuse BLOCKSIZE:10240 > 20170823000017 slot-7 reuse BLOCKSIZE:10240 > 20170822000017 slot-6 reuse BLOCKSIZE:10240 > 20170821000017 slot-5 reuse BLOCKSIZE:10240 > 20170820000017 slot-4 reuse BLOCKSIZE:10240 > 20170819000017 slot-3 reuse BLOCKSIZE:10240 > 20170818000017 slot-2 reuse BLOCKSIZE:10240 > 20170817000017 slot-1 reuse BLOCKSIZE:10240 > 20170816000017 slot-15 reuse BLOCKSIZE:10240 > 20170815000017 slot-14 reuse BLOCKSIZE:1024 > Do you have a lifecycle policy on the bucket? > > After a backup, you can do: amcheckdump CONF > to verify amanda can read the backup. > > You can also get the list of all s3 objects and check if something was > removed. > > Jean-Louis > > > On 29/08/17 08:00 AM, Giorgio Valoti wrote: >> >> > Il giorno 29 ago 2017, alle ore 13:55, Jean-Louis Martineau >> > <jmartin...@carbonite.com> ha scritto: >> > >> > The list of files with the 'slot-bb2e601e63b3408bb6a865e442a28366-6' >> > prefix (slot-6) show there is no data in the slot, only the tape label. >> > There is data only on slot-10,11,12 and 13. >> > I have no idea why there is no data in the others slots. >> > I assume the data was there after the backup, but I don't how they got >> > erased. >> >> As far as I can tell, nothing touches this bucket except Amanda. What should >> I check or keep checked to avoid issues like this? Do you think should I >> change / check something in my conf? >> >> > >> > Jean-Louis >> > >> > On 29/08/17 07:31 AM, Giorgio Valoti wrote: >> >> >> >> > Il giorno 29 ago 2017, alle ore 12:36, Jean-Louis Martineau >> >> > <jmartin...@carbonite.com> ha scritto: >> >> > >> >> > This error often means that some s3 objects are missing. >> >> > >> >> > Do some of the data was restored? >> >> > Can you post the amfetchdump debug file? >> >> >> >> Here it is: >> >> >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: pid 26587 ruid 34 >> >> euid 34 version 3.3.6: start at Tue Aug 29 08:27:19 2017 >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: Arguments: -p -a >> >> zfs-dataset ci-master rpool/jenkins 20170822000017 >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: pid 26587 ruid 34 >> >> euid 34 version 3.3.6: rename at Tue Aug 29 08:27:19 2017 >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: Using state file: >> >> /var/lib/amanda/state/s3-statefile >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: >> >> Amanda::Recovery::Clerk: loading volume 'slot-6' >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: find_volume labeled >> >> 'slot-6' >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: parse_inventory: >> >> load slot 6 with label 'slot-6' >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: S3 driver using >> >> bucket 'onebip-backups-admin', prefix >> >> 'slot-bb2e601e63b3408bb6a865e442a28366-6' >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: curl version: >> >> libcurl/7.47.0 OpenSSL/1.0.2g zlib/1.2.8 libidn/1.32 librtmp/2.3 >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: curl compiled for >> >> OPENSSL >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: Create 8 threads >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: GET >> >> https://onebip-backups-admin.s3.amazonaws.com/slot-bb2e601e63b3408bb6a865e442a28366-6f00000003-filestartfailed >> >> with 404/NoSuchKey >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: GET >> >> https://onebip-backups-admin.s3.amazonaws.com/slot-bb2e601e63b3408bb6a865e442a28366-6f00000002-filestartfailed >> >> with 404/NoSuchKey >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: Device >> >> s3:onebip-backups-admin/slot-bb2e601e63b3408bb6a865e442a28366-6 error = >> >> 'Attempt to read past tape-end file' >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: FAILURE: Attempt to >> >> read past tape-end file >> >> Tue Aug 29 08:27:19 2017: thd-0x1a8a600: amfetchdump: pid 26587 finish >> >> time Tue Aug 29 08:27:19 2017 >> >> >> >> >> >> >> >> > Can you get a list of all objects in the S3 bucket? >> >> >> >> >> >> Here it is. I have scraped the web page and yes the key is missing but I >> >> don’t know why. I mean, I receive the reports every day and they seemed >> >> OK to me. Let me know if you need my conf, as well, thank you. >> >> >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000001-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000001-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000002-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000002-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000001.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000002.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000003.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000004.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000005.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000006.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000007.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000008.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000009.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b000000000000000a.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b000000000000000b.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b000000000000000c.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b000000000000000d.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b000000000000000e.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b000000000000000f.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000010.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000011.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000012.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000013.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000014.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000015.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000016.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000017.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000018.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b0000000000000019.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-b000000000000001a.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000003-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000004-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000004-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000005-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-10f00000005-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-10special-tapestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000001-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000001-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000002-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000002-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000003-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000003-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000004-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000004-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000005-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-11f00000005-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-11special-tapestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000001-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000001-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000002-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000002-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000003-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000003-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000004-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000004-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000005-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-12f00000005-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-12special-tapestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000001-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000001-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000002-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000002-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000001.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000002.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000003.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000004.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000005.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000006.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000007.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000008.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000009.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000000a.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000000b.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000000c.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000000d.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000000e.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000000f.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000010.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000011.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000012.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000013.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000014.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000015.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000016.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000017.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000018.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000019.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000001a.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000001b.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000001c.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000001d.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000001e.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b000000000000001f.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000020.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000021.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-b0000000000000022.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000003-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000004-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000004-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000005-b0000000000000000.data >> >> slot-bb2e601e63b3408bb6a865e442a28366-13f00000005-filestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-13special-tapestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-14special-tapestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-15special-tapestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-1special-tapestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-2special-tapestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-3special-tapestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-4special-tapestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-5special-tapestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-6special-tapestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-7special-tapestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-8special-tapestart >> >> slot-bb2e601e63b3408bb6a865e442a28366-9special-tapestart >> >> >> >> >> >> >> >> >> >> >> >> > >> >> > Jean-Louis >> >> > >> >> > On 29/08/17 05:22 AM, Giorgio Valoti wrote: >> >> > > Hi all, I’m testing the restore process with Amanda and I’m getting >> >> > > this error with amfetchdump: >> >> > > >> >> > > amfetchdump -p -a zfs-dataset ci-master rpool/jenkins 20170822000017 >> >> > > >> >> > >> 1 volume(s) needed for restoration >> >> > >> The following volumes are needed: slot-6 >> >> > >> ERROR: Attempt to read past tape-end file >> >> > > And here is the list of the relevant backups: >> >> > > >> >> > >> backup@backup-server:~$ amadmin zfs-dataset find --sort D ci-master >> >> > >> rpool/jenkins >> >> > >> >> >> > >> date host disk lv tape or file file part status >> >> > >> 2017-08-29 00:00:17 ci-master rpool/jenkins 2 slot-13 3 1/1 OK >> >> > >> 2017-08-28 00:00:17 ci-master rpool/jenkins 2 slot-12 1 1/1 OK >> >> > >> 2017-08-27 00:00:17 ci-master rpool/jenkins 2 slot-11 1 1/1 OK >> >> > >> 2017-08-26 00:00:17 ci-master rpool/jenkins 1 slot-10 3 1/1 OK >> >> > >> 2017-08-25 00:00:17 ci-master rpool/jenkins 1 slot-9 3 1/1 OK >> >> > >> 2017-08-24 00:00:17 ci-master rpool/jenkins 1 slot-8 3 1/1 OK >> >> > >> 2017-08-23 00:00:17 ci-master rpool/jenkins 1 slot-7 3 1/1 OK >> >> > >> 2017-08-22 00:00:17 ci-master rpool/jenkins 0 slot-6 3 1/1 OK >> >> > >> 2017-08-21 00:00:17 ci-master rpool/jenkins 2 slot-5 1 1/1 OK >> >> > >> 2017-08-20 00:00:17 ci-master rpool/jenkins 2 slot-4 1 1/1 OK >> >> > >> 2017-08-19 00:00:17 ci-master rpool/jenkins 2 slot-3 1 1/1 OK >> >> > >> 2017-08-18 00:00:17 ci-master rpool/jenkins 2 slot-2 1 1/1 OK >> >> > >> 2017-08-17 00:00:17 ci-master rpool/jenkins 1 slot-1 3 1/1 OK >> >> > >> 2017-08-16 00:00:17 ci-master rpool/jenkins 1 slot-15 1 1/1 OK >> >> > >> 2017-08-15 00:00:17 ci-master rpool/jenkins 1 slot-14 1 1/1 OK >> >> > > In other words, I’m trying to fetch the level 0 backup. Also, it >> >> > > should be noted that I’m using “tapes” with other Amanda clients as >> >> > > well.: >> >> > > >> >> > >> backup@backup-server:~$ amadmin zfs-dataset find --sort D | grep >> >> > >> slot-6 >> >> > >> 2017-08-22 00:00:17 backup-server rpool/amanda-conf 1 slot-6 5 1/1 OK >> >> > >> 2017-08-22 00:00:17 backup-server rpool/amanda-state 1 slot-6 4 1/1 >> >> > >> OK >> >> > >> 2017-08-22 00:00:17 ci-master rpool/jenkins 0 slot-6 3 1/1 OK >> >> > >> 2017-08-22 00:00:17 vpn-server rpool/openvpn 1 slot-6 2 1/1 OK >> >> > >> 2017-08-22 00:00:17 backup-server rpool/amanda-home 1 slot-6 1 1/1 OK >> >> > > I’m sure I’m missing something here, because this is a real bummer. >> >> > > Anyone on this list willing to help me, please? >> >> > > >> >> > > >> >> > > >> >> > > -- >> >> > > Giorgio Valoti >> >> > > >> >> > > >> >> > >> >> > >> >> > Disclaimer >> >> > >> >> > This message is the property of CARBONITE, INC. and may contain >> >> > confidential or privileged information. >> >> > >> >> > If this message has been delivered to you by mistake, then do not copy >> >> > or deliver this message to anyone. Instead, destroy it and notify me by >> >> > reply e-mail. >> >> > >> >> >> >> >> >> -- >> >> Giorgio Valoti >> > >> > >> > >> > >> > Disclaimer >> > >> > This message is the property of CARBONITE, INC. and may contain >> > confidential or privileged information. >> > >> > If this message has been delivered to you by mistake, then do not copy or >> > deliver this message to anyone. Instead, destroy it and notify me by reply >> > e-mail. >> > > > > > > Disclaimer > > This message is the property of CARBONITE, INC. and may contain confidential > or privileged information. > > If this message has been delivered to you by mistake, then do not copy or > deliver this message to anyone. Instead, destroy it and notify me by reply > e-mail. >