Hi again.
So, as expected, the warning occurred again when job 94511 was copied
(but also two new warnings for this specific job):
Warning: dird/catreq.cc:608 MD5 digest not same File=3 as attributes=2
Warning: dird/catreq.cc:608 MD5 digest not same File=18 as attributes=17
Warning: dird/catreq.cc:608 MD5 digest not same File=23 as attributes=22
The restore of the copy also errored in the same manner as before:
2024-02-26 10:43:50 bareos-fd JobId 95784: Error:
findlib/attribs.cc:381 File size of restored file
/tmp/bareos-restores/var/backup/old/2024-02-07_23:30/backup_www_1100CC.tar.gz
not correct. Original 738212473, restored 442105856.
2024-02-26 10:44:10 bareos-fd JobId 95784: Error:
findlib/attribs.cc:381 File size of restored file
/tmp/bareos-restores/var/backup/old/2024-02-12_23:30/backup_www_1100CC.tar.gz
not correct. Original 856739265, restored 783548416.
2024-02-26 10:44:17 bareos-fd JobId 95784: Error:
findlib/attribs.cc:381 File size of restored file
/tmp/bareos-restores/var/backup/old/2024-02-14_23:30/backup_www_1100CC.tar.gz
not correct. Original 856739984, restored 31064064.
For now I'll try disabling autoxflate and see if that solves the issue.
Best regards,
Samuel
On Friday, February 23, 2024 at 2:02:28 PM UTC+1 Samuel wrote:
Thanks for looking into it!
Hm, there seems to be no file with fileindex=3 in the database for
the copy:
Enter SQL query: select * from file where jobid=94146 and fileindex=3;
No results to list.
All files that are being backed up are just regular files, nothing
special.
The copy runs locally on the director's host from its SD to the
same SD.
Here's file 2 and 3 of today's consolidated incremental backup
which will be copied tomorrow and most likely lead to the same
warning again as it has for the last few days (for this specific
fileset it always warns about `File=3 attributes=2`):
Enter SQL query: select *, decode_lstat(lstat) from file where
jobid=94511 and fileindex=2;
+-------------+-----------+-------+---------+----------+--------+--------+--------+----------------------------------------------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------+
| fileid | fileindex | jobid | pathid | deltaseq | markid |
fhinfo | fhnode | lstat | md5 |
name | decode_lstat |
+-------------+-----------+-------+---------+----------+--------+--------+--------+----------------------------------------------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------+
| 484,575,603 | 2 | 94511 | 910,901 | 0 | 0 |
0 | 0 | P0D CgA4 IGk B A i A sADp5 BAA FgAo BlxAt1
BlxASF BlxVYH A A d | D/VouduY5TF4KFacjRE7Hw |
backup_www_1100CC.tar.gz |
(64771,655416,33188,1,0,34,0,738212473,4096,1441832,1707346805,1707345029,1707431431,0,0,29)
|
+-------------+-----------+-------+---------+----------+--------+--------+--------+----------------------------------------------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------+
Enter SQL query: select *, decode_lstat(lstat) from file where
jobid=94511 and fileindex=3;
+-------------+-----------+-------+---------+----------+--------+--------+--------+---------------------------------------------------------------+------------------------+-----------------------------+-------------------------------------------------------------------------------------------+
| fileid | fileindex | jobid | pathid | deltaseq | markid |
fhinfo | fhnode | lstat | md5 |
name | decode_lstat |
+-------------+-----------+-------+---------+----------+--------+--------+--------+---------------------------------------------------------------+------------------------+-----------------------------+-------------------------------------------------------------------------------------------+
| 484,575,604 | 3 | 94511 | 910,901 | 0 | 0 |
0 | 0 | P0D CgA5 IGk B A i A BA9gA BAA IHw BlxAt8 BlxASH
BlxVYH A A d | J9h+OdR/6XCqhqpJdJvYtw |
backup_databases_1100CC.tar |
(64771,655417,33188,1,0,34,0,17029120,4096,33264,1707346812,1707345031,1707431431,0,0,29)
|
+-------------+-----------+-------+---------+----------+--------+--------+--------+---------------------------------------------------------------+------------------------+-----------------------------+-------------------------------------------------------------------------------------------+
Restore of this job (jobid=94511) works just fine:
23-Feb 13:44 bareos-sd JobId 94559: Releasing device
"FileDevice-ReadOnly-0005" (/backup_1/bareos).
23-Feb 13:44 bareos-dir JobId 94559: Max configured use
duration=82,800 sec. exceeded. Marking Volume
"AI-Consolidated-1771" as Used.
23-Feb 13:44 bareos-dir JobId 94559: Bareos bareos-dir
23.0.2~pre32.0a0e55739 (31Jan24):
Build OS: Ubuntu 20.04.5 LTS
JobId: 94559
Job: Restore-Files.2024-02-23_13.43.47_38
Restore Client: "bareos-fd" 23.0.2~pre32.0a0e55739
(31Jan24) Ubuntu 20.04.5 LTS,ubuntu
Start time: 23-Feb-2024 13:43:49
End time: 23-Feb-2024 13:44:20
Elapsed time: 31 secs
Files Expected: 33
Files Restored: 33
Bytes Restored: 9,143,514,060
Rate: 294952.1 KB/s
FD Errors: 0
FD termination status: OK
SD termination status: OK
Bareos binary info: Bareos community build (UNSUPPORTED):
Get professional support from https://www.bareos.com
Job triggered by: User
Termination: Restore OK
I'll also try restoring its copy from tape once it has been copied.
Best regards,
Samuel
On Friday, February 23, 2024 at 1:16:32 PM UTC+1 Sebastian Sura wrote:
I meant to say File 3 has no stream 1, sorry for the confusion!
Am 23.02.24 um 13:15 schrieb Sebastian Sura:
Hi Samuel
thanks for the gathering this info. Ill look into the bscan
issue as well though i think this might be a known issue.
Regardless, regarding your actual issue: The bscan output
shows that File 3 is missing its attributes (it has no stream 3).
This confuses the director as it never got told that file 3
now started getting backed up and this is why you get the warning
message from the director.
The same is happening during the restore: the filedaemon
never got told that now a new file was started (because it
never got the attribute stream)
so it basically merged both File 2 and File 3 into one file.
Even with this in mind, i think the restore should have
caught that and issues a warning.
Ill look into why this did not happen.
If you add up all the data records (stream=2) with fileid=2
and fileid=3, you will get 738212473 bytes, which is exactly
what the filedaemon reported
as size for file 2. Since the log contains the size it
expected i imagine that you could manually restore file 3 by
splitting file 2 into two.
Can you check which file fileid 3 corresponds to ? Is it a
special kind of file or just a normal one ?
Ill try to see if i can reproduce your issue in the copy
system test. Do you do a local copy (so copy to the same sd)
or a remote one?
Kind Regards
Sebastian Sura
Am 23.02.24 um 12:56 schrieb 'Samuel' via bareos-users:
Hi Sebastian,
After looking at the logs some more I noticed that all jobs
with this warning have in common that they're using
compression (LZ4) and autoxflate on replication.
Perhaps this combination is still not entirely fixed yet.
I'll try disabling autoxflate in the coming days.
The entire result of bscan is in the attachment. However
here's the last part of it where bscan seems to abort (?):
$ sudo -u bareos bscan -b md5_digest_error_copy_only.bsr
--list-records TapeDevice2 2>&1 | tee records.txt
...
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=23 Stream=2 len=65536
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=23 Stream=2 len=65536
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=23 Stream=2 len=65536
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=23 Stream=2 len=22528
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=23 Stream=3 len=16
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=24 Stream=1 len=100
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=25 Stream=1 len=83
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=-5 Stream=94145 len=193
bscan: stored/bscan.cc:681-0 Could not find SessId=3480
SessTime=1707222936 for EOS record.
Records would have been added or updated in the catalog:
0 Media
1 Pool
0 Job
0 File
0 RestoreObject
23-Feb 12:05 bscan JobId 0: Releasing device "TapeDevice2"
(/dev/tape/by-id/scsi-35000e111c71ac0bf-nst).
I also tried restoring the copy which ends in error.
Most files are restored successfully except for one file
which happens to have FileIndex=2 (corresponding to the
`attributes=2` in the warning?):
Enter SQL query: select path,name from file,path where
jobid=94146 and fileindex=2 and file.pathid=path.pathid;
+-----------------------------------+--------------------------+
| path | name |
+-----------------------------------+--------------------------+
| /var/backup/old/2024-02-07_23:30/ | backup_www_1100CC.tar.gz |
+-----------------------------------+--------------------------+
A full restore of the original non-copy backup jobs works
fine, as do restores of other copy jobs on tape of filesets
that don't use compression.
The original job that was copied (93682) doesn't exist
anymore; I think it was an always-incremental consolidated
incremental backup which got consolidated again the
following day.
Best regards,
Samuel
--
You received this message because you are subscribed to the
Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/bareos-users/e766e8c6-5b58-424f-816f-f7c5d7252180n%40googlegroups.com
<https://groups.google.com/d/msgid/bareos-users/e766e8c6-5b58-424f-816f-f7c5d7252180n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
Sebastian [email protected]
Bareos GmbH & Co. KG Phone: +49 221 630693-0
https://www.bareos.com
Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz
--
You received this message because you are subscribed to the
Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/bareos-users/111cecfe-d3e1-4eb9-8a0e-f053a2777d0f%40bareos.com
<https://groups.google.com/d/msgid/bareos-users/111cecfe-d3e1-4eb9-8a0e-f053a2777d0f%40bareos.com?utm_medium=email&utm_source=footer>.
--
Sebastian [email protected]
Bareos GmbH & Co. KG Phone: +49 221 630693-0
https://www.bareos.com
Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz
--
You received this message because you are subscribed to the Google
Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/bareos-users/3823a22d-b49e-4892-8b45-6c932ce7e699n%40googlegroups.com
<https://groups.google.com/d/msgid/bareos-users/3823a22d-b49e-4892-8b45-6c932ce7e699n%40googlegroups.com?utm_medium=email&utm_source=footer>.
