Re: [Bacula-users] verify error with LTO hardware encryption SOLVED

[Mark D. Strohm]

Hello Kern-

Thanks for the reply.  Something like that may be at the root of the problem, 
but the drive does not report being in SysV mode.  It shows only these options 
set: buffer-writes, async-writes, read-ahead, can-bsr and scsi2logical.

The Tape Testing Tools and their debug modes were very helpful for working 
through the problem.  Whatever it is seems to be very specific to enabling 
hardware encryption on my IBM LTO-4 SCSI drives.  They’re fine with encryption 
off, and the HP LTO-5 and 6 SAS drives I have work fine with encryption on or 
off.

-Mark-

On 12/22/2015 17:26, Kern Sibbald wrote:
> Hello,
> 
> In looking at your patch, I would guess that your tape drive is set in
> SysV mode, which Bacula does not support.  Bacula only supports Berkeley
> tape conventions. 
> 
> Have you worked through the Tape Testing chapter of the main manual?  If
> you have, I would find it very strange that the testing did not pick up
> the problem.
> 
> Best regards,
> Kern



--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] verify error with LTO hardware encryption SOLVED

[Kern Sibbald]

Hello,

In looking at your patch, I would guess that your tape drive is set in
SysV mode, which Bacula does not support.  Bacula only supports Berkeley
tape conventions. 

Have you worked through the Tape Testing chapter of the main manual?  If
you have, I would find it very strange that the testing did not pick up
the problem.

Best regards,
Kern

On 12/21/2015 11:19 PM, Mark D. Strohm wrote:
> Just to follow up, I did find a workaround for the I/O errors with LTO 
> hardware encryption:  At the end of a tape file, space a record forward, then 
> a record back (code below).
>
> I have not found the actual cause of the problem.  It is most likely with the 
> vendor firmware for that particular IBM LTO-4 model.  But the workaround does 
> permit both verifies and restores to work properly with encrypted tapes.
>
> -Mark-
>
>
> bacula-7.0.5
>
> *** block.c.orig  2014-07-29 09:31:22.0 -0700
> --- block.c   2015-11-13 14:16:04.0 -0800
> ***
> *** 441,446 
> --- 441,452 
> } else {
>Mmsg3(dev->errmsg, _("Read zero bytes Vol=%s at %lld on device 
> %s.\n"),
>  dev->VolCatInfo.VolCatName, pos, dev->print_name());
> +  if (dev->fsr(1)) {/* to deal with the encrypted eof jams, 
> forward space one record */
> + dev->bsr(1);   /* if that works without error, back space 
> into position */
> +  } else {  /* if it throws an error ... */
> + dev->set_ateof();  /* set the ateof flag to trigger the eom 
> detection below */
> + dev->file--;   /* and reset the file number */
> +  }
> }
> dev->block_num = 0;
> block->read_len = 0;
>
>
>
> --
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>



--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] verify error with LTO hardware encryption SOLVED

[Mark D. Strohm]

Just to follow up, I did find a workaround for the I/O errors with LTO hardware 
encryption:  At the end of a tape file, space a record forward, then a record 
back (code below).

I have not found the actual cause of the problem.  It is most likely with the 
vendor firmware for that particular IBM LTO-4 model.  But the workaround does 
permit both verifies and restores to work properly with encrypted tapes.

-Mark-


bacula-7.0.5

*** block.c.orig2014-07-29 09:31:22.0 -0700
--- block.c 2015-11-13 14:16:04.0 -0800
***
*** 441,446 
--- 441,452 
} else {
   Mmsg3(dev->errmsg, _("Read zero bytes Vol=%s at %lld on device 
%s.\n"),
 dev->VolCatInfo.VolCatName, pos, dev->print_name());
+  if (dev->fsr(1)) {/* to deal with the encrypted eof jams, 
forward space one record */
+ dev->bsr(1);   /* if that works without error, back space 
into position */
+  } else {  /* if it throws an error ... */
+ dev->set_ateof();  /* set the ateof flag to trigger the eom 
detection below */
+ dev->file--;   /* and reset the file number */
+  }
}
dev->block_num = 0;
block->read_len = 0;



--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] verify error with LTO hardware encryption

[Martin Simmons]

> On Tue, 20 Oct 2015 15:02:23 -0700, Mark D Strohm said:
> 
> Hello Patti and Martin-
> 
> Thank you for the replies.
> 
> As I understand it, Bacula should have no problem with LTO hardware 
> encryption because (once set) it is supposed to be transparent at the user 
> level, the same as hardware compression.  But apparently it is not perfectly 
> transparent.  
> 
> I’m bringing the problem up on the Bacula list because I’m confident it can 
> be worked around in software (mt and dd relate to the encrypted tape 
> successfully), and changing the drive firmware isn’t really an option.  But 
> I’m not sure whether a simple configuration change is needed, or if I have to 
> modify something in the Bacula programs.
> 
> stenc does appear to be configuring the drive encryption properly.  It can 
> move the drive in-to and out-of all the modes I intend to use.  It correctly 
> reports the Key-label active in the drive, and reports mismatches with the 
> Key-label on the tape.
> 
> The system has been working without encryption for several months, using the 
> mptspi and st Driver Modules.
> 
> The system log shows only activity after the error.  The Storage Daemon stops 
> working actively when it reaches the second file, and reports an error after 
> about ten minutes.  Then there are log entries as the driver attempts to 
> abort a task, fails, and resets the device.
> 
> Oct 19 14:19:43 ccnback kernel: mptscsih: ioc0: attempting task abort! 
> (sc=8801b7682380)
> Oct 19 14:19:43 ccnback kernel: st 6:0:3:0: CDB: 
> Oct 19 14:19:43 ccnback kernel: Read(6): 08 00 00 fc 00 00
> Oct 19 14:19:43 ccnback kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) 
> (sc=8801b7682380)
> Oct 19 14:19:43 ccnback kernel: mptscsih: ioc0: attempting target reset! 
> (sc=8801b7682380)
> Oct 19 14:19:43 ccnback kernel: st 6:0:3:0: CDB: 
> Oct 19 14:19:43 ccnback kernel: Read(6): 08 00 00 fc 00 00
> Oct 19 14:19:53 ccnback kernel: mptscsih: ioc0: WARNING - Issuing Reset from 
> mptscsih_IssueTaskMgmt!! doorbell=0x2400
> Oct 19 14:19:54 ccnback kernel: mptscsih: ioc0: target reset: SUCCESS 
> (sc=8801b7682380)
> Oct 19 14:19:54 ccnback kernel: st0: Error 8 (driver bt 0x0, host bt 0x8).
> Oct 19 14:19:54 ccnback kernel: scsi target6:0:3: Beginning Domain Validation
> Oct 19 14:19:54 ccnback kernel: scsi target6:0:3: Ending Domain Validation
> Oct 19 14:19:54 ccnback kernel: scsi target6:0:3: FAST-80 WIDE SCSI 160.0 
> MB/s DT (12.5 ns, offset 127)
> Oct 19 14:19:55 ccnback kernel: scsi target6:0:3: Beginning Domain Validation
> Oct 19 14:19:57 ccnback kernel: scsi target6:0:3: Ending Domain Validation
> Oct 19 14:19:57 ccnback kernel: scsi target6:0:3: FAST-80 WIDE SCSI 160.0 
> MB/s DT (12.5 ns, offset 127)

I'm not an expert on this, but a problem that requires the driver to issue a
target reset suggests a bug in the driver or in the drive's firmware.

Have you tried smartctl or tapeinfo to get more information about the failure
from the drive (e.g. TapeAlert messages)?

__Martin

--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] verify error with LTO hardware encryption

[Clark, Patti]

Mark,

Bacula is not involved in hardware (tape drive) encryption.

The encryption dialog is an exchange of key information between the drive
and the encryption key manager, in your case stenc.  If your tapes were
initially written to prior to using the encryption capability, the tapes
can never be hardware encrypted.  The opposite is also true, if you write
to the tapes the very first time using encryption, they can never be
written to unencrypted.

What testing did you perform prior to getting bacula involved?  I'm not
familiar with stenc, are you sure that it is conversing properly with your
hardware?

Patti Clark
Linux System Administrator
R Systems Support Oak Ridge National Laboratory


On 10/19/15, 6:47 PM, "Mark D. Strohm"  wrote:

>Hello-
>
>Is there a trick to using Bacula with LTO hardware encryption enabled?
>
>With drive encryption turned on, verify jobs are hitting an I/O error
>reading the first record of a tape file.
>
>On a test job that went to files 78, 79 and 80, the error looks like this:
>
>19-Oct 14:03 ccnback-sd JobId 988: Ready to read from volume "CCNB12" on
>tape device "Magnum-224-LTO4"
>(/dev/tape/by-id/scsi-1IBM_ULTRIUM-TD4_1310025811-nst).
>19-Oct 14:03 ccnback-sd JobId 988: Forward spacing Volume "CCNB12" to
>file:block 78:0.
>19-Oct 14:19 ccnback-sd JobId 988: Error: block.c:429 Read error on fd=5
>at file:blk 79:0 on device "Magnum-224-LTO4"
>(/dev/tape/by-id/scsi-1IBM_ULTRIUM-TD4_1310025811-nst). ERR=Input/output
>error.
>19-Oct 14:19 ccnback-sd JobId 988: End of Volume at file 79 on device
>"Magnum-224-LTO4" (/dev/tape/by-id/scsi-1IBM_ULTRIUM-TD4_1310025811-nst),
>Volume ³CCNB12"
>
>dd can read the data from both files.  The boundary is on the phrase
>"varius sed feugiat².  The end of file 78, and the start of 79 are:
>
>"varius sed "
>
>"~\274\301\214^@^@\374^@^@^A.\300BB02^@^@^@^AV%X\226^@^@^@3\377\377\377\37
>6^@^@\313\233feugiat²
>
>I¹m using Bacula 7.0.5 with an LTO-4 drive and stenc 1.0.7 to control
>encryption.
>
>Any advice would be appreciated.
>
>Thank you very much.
>Mark
>
>
>--
>
>___
>Bacula-users mailing list
>Bacula-users@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/bacula-users


--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] verify error with LTO hardware encryption

[Martin Simmons]

> On Mon, 19 Oct 2015 15:47:01 -0700, Mark D Strohm said:
> 
> Hello-
> 
> Is there a trick to using Bacula with LTO hardware encryption enabled?
> 
> With drive encryption turned on, verify jobs are hitting an I/O error reading 
> the first record of a tape file.
> 
> On a test job that went to files 78, 79 and 80, the error looks like this:
> 
> 19-Oct 14:03 ccnback-sd JobId 988: Ready to read from volume "CCNB12" on tape 
> device "Magnum-224-LTO4" 
> (/dev/tape/by-id/scsi-1IBM_ULTRIUM-TD4_1310025811-nst).
> 19-Oct 14:03 ccnback-sd JobId 988: Forward spacing Volume "CCNB12" to 
> file:block 78:0.
> 19-Oct 14:19 ccnback-sd JobId 988: Error: block.c:429 Read error on fd=5 at 
> file:blk 79:0 on device "Magnum-224-LTO4" 
> (/dev/tape/by-id/scsi-1IBM_ULTRIUM-TD4_1310025811-nst). ERR=Input/output 
> error.
> 19-Oct 14:19 ccnback-sd JobId 988: End of Volume at file 79 on device 
> "Magnum-224-LTO4" (/dev/tape/by-id/scsi-1IBM_ULTRIUM-TD4_1310025811-nst), 
> Volume “CCNB12"
> 
> dd can read the data from both files.  The boundary is on the phrase "varius 
> sed feugiat”.  The end of file 78, and the start of 79 are:
> 
> "varius sed "
> 
> "~\274\301\214^@^@\374^@^@^A.\300BB02^@^@^@^AV%X\226^@^@^@3\377\377\377\376^@^@\313\233feugiat”
> 
> I’m using Bacula 7.0.5 with an LTO-4 drive and stenc 1.0.7 to control 
> encryption.
> 
> Any advice would be appreciated.

Have you checked the syslog in case it recorded something more precise about
the I/O error?

Also, which tape device driver are you using?  In the past, people have had
problems with lin_tape (use st instead).

__Martin

--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] verify error with LTO hardware encryption

[Mark D. Strohm]

Hello Patti and Martin-

Thank you for the replies.

As I understand it, Bacula should have no problem with LTO hardware encryption 
because (once set) it is supposed to be transparent at the user level, the same 
as hardware compression.  But apparently it is not perfectly transparent.  

I’m bringing the problem up on the Bacula list because I’m confident it can be 
worked around in software (mt and dd relate to the encrypted tape 
successfully), and changing the drive firmware isn’t really an option.  But I’m 
not sure whether a simple configuration change is needed, or if I have to 
modify something in the Bacula programs.

stenc does appear to be configuring the drive encryption properly.  It can move 
the drive in-to and out-of all the modes I intend to use.  It correctly reports 
the Key-label active in the drive, and reports mismatches with the Key-label on 
the tape.

The system has been working without encryption for several months, using the 
mptspi and st Driver Modules.

The system log shows only activity after the error.  The Storage Daemon stops 
working actively when it reaches the second file, and reports an error after 
about ten minutes.  Then there are log entries as the driver attempts to abort 
a task, fails, and resets the device.

Oct 19 14:19:43 ccnback kernel: mptscsih: ioc0: attempting task abort! 
(sc=8801b7682380)
Oct 19 14:19:43 ccnback kernel: st 6:0:3:0: CDB: 
Oct 19 14:19:43 ccnback kernel: Read(6): 08 00 00 fc 00 00
Oct 19 14:19:43 ccnback kernel: mptscsih: ioc0: task abort: FAILED (rv=2003) 
(sc=8801b7682380)
Oct 19 14:19:43 ccnback kernel: mptscsih: ioc0: attempting target reset! 
(sc=8801b7682380)
Oct 19 14:19:43 ccnback kernel: st 6:0:3:0: CDB: 
Oct 19 14:19:43 ccnback kernel: Read(6): 08 00 00 fc 00 00
Oct 19 14:19:53 ccnback kernel: mptscsih: ioc0: WARNING - Issuing Reset from 
mptscsih_IssueTaskMgmt!! doorbell=0x2400
Oct 19 14:19:54 ccnback kernel: mptscsih: ioc0: target reset: SUCCESS 
(sc=8801b7682380)
Oct 19 14:19:54 ccnback kernel: st0: Error 8 (driver bt 0x0, host bt 0x8).
Oct 19 14:19:54 ccnback kernel: scsi target6:0:3: Beginning Domain Validation
Oct 19 14:19:54 ccnback kernel: scsi target6:0:3: Ending Domain Validation
Oct 19 14:19:54 ccnback kernel: scsi target6:0:3: FAST-80 WIDE SCSI 160.0 MB/s 
DT (12.5 ns, offset 127)
Oct 19 14:19:55 ccnback kernel: scsi target6:0:3: Beginning Domain Validation
Oct 19 14:19:57 ccnback kernel: scsi target6:0:3: Ending Domain Validation
Oct 19 14:19:57 ccnback kernel: scsi target6:0:3: FAST-80 WIDE SCSI 160.0 MB/s 
DT (12.5 ns, offset 127)


Thanks again.
Mark


--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] verify error with LTO hardware encryption

[Mark D. Strohm]

Hello-

Is there a trick to using Bacula with LTO hardware encryption enabled?

With drive encryption turned on, verify jobs are hitting an I/O error reading 
the first record of a tape file.

On a test job that went to files 78, 79 and 80, the error looks like this:

19-Oct 14:03 ccnback-sd JobId 988: Ready to read from volume "CCNB12" on tape 
device "Magnum-224-LTO4" (/dev/tape/by-id/scsi-1IBM_ULTRIUM-TD4_1310025811-nst).
19-Oct 14:03 ccnback-sd JobId 988: Forward spacing Volume "CCNB12" to 
file:block 78:0.
19-Oct 14:19 ccnback-sd JobId 988: Error: block.c:429 Read error on fd=5 at 
file:blk 79:0 on device "Magnum-224-LTO4" 
(/dev/tape/by-id/scsi-1IBM_ULTRIUM-TD4_1310025811-nst). ERR=Input/output error.
19-Oct 14:19 ccnback-sd JobId 988: End of Volume at file 79 on device 
"Magnum-224-LTO4" (/dev/tape/by-id/scsi-1IBM_ULTRIUM-TD4_1310025811-nst), 
Volume “CCNB12"

dd can read the data from both files.  The boundary is on the phrase "varius 
sed feugiat”.  The end of file 78, and the start of 79 are:

"varius sed "

"~\274\301\214^@^@\374^@^@^A.\300BB02^@^@^@^AV%X\226^@^@^@3\377\377\377\376^@^@\313\233feugiat”

I’m using Bacula 7.0.5 with an LTO-4 drive and stenc 1.0.7 to control 
encryption.

Any advice would be appreciated.

Thank you very much.
Mark


--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users