On 22/12/16 17:49, Drew Von Spreecken wrote:
> Thank you Alan for the detailed response.
>
> I was unaware of the GPL violations, this is an issue for me and it
> will be removed promptly.
>
> The block size I am aiming for was 512KB-1MB and have done extensive
> research on tuning. This should also of course be within specification
> of bacula-sd.
500k to 1Mb seems to be about the point where the gain ends.
>
> Because I am currently writing at a 64KB block, I get around 70MB/s of
> throughput on a mix of compressible and non-compressible data. Even
> changing it to a working 256KB improves what I am writing by about
> 15MB/s. Changing to any other value causes an immediate Device Or
> Resource busy error.
>
What parameter are you using and where in the config file are you using it?
> The setup has worked well for years until my storage array doubled in
> size. It is taking over a week to do a backup.
>
> This is direct-attached storage so I don't believe the network buffer
> should have an effect, right?
>
Direct attached to what? Do you mean your disks, tape drives and bacula
are all running on the same system? If so, then you're probably right.
> I'm not sure I completely understand this statement.
>
> "
>
> You cannot change maximum block sizes on a single volume (Tape). Doing
> so is extremely likely to result in an unreadable volume past the point
> where the block size has changed (reading older tapes with smaller
> maximum block sizes is still ok)
>
> "
>
> My plan was to overwrite all tapes that may have a smaller block size.
> It doesn't matter at this time if I temporarily don't have a tape
> backup and I have no old archives I am concerned with.
You don't need to do that. An older tape will smaller block size will
read just fine, but don't mix block sizes on any one tape.
>
> Rewinding the tape and writing an EOF marker and rewinding again
> should solve this correct?
>
Probably
> To me it seems like a the SAS controller I am connected to but it just
> seems so unlikely it isn't able to pass blocks greater than 256KB but
> I have no way of verifying this.
>
What's your SAS controller chipset? Some of the older ones are downright
awful.
> The more I write this the more I am convincing myself it probably has
> nothing to do with Bacula (rip-off version) or the configuration. I
> was just hoping someone has run into something similar before.
>
> Regards,
>
> drewv
>
>
>
> On 12/22/2016 11:15 AM, Alan Brown wrote:
>> On 22/12/16 14:31, Drew Von Spreecken wrote:
>>> Greetings,
>>>
>>> I have run into an issue and am looking for input. I have a SAS tape
>>> autoloader with an IBM HH-LTO6 drive running the newest firmware.
>>> It is currently connected to a Adaptec 78165 HBA/Raid controller via
>>> SAS.
>> The maximum block size for IBM LTO6 drives (HH or full height) is 8MB
>> (It's usually 16MB for HP drives)
>>
>>> The issue I am having is when I attempt to modify the block-size to
>>> anything over 256KB that BareOS writes to tape, it fails. To simplify
>>> troubleshooting I have opted to use a combination of btape and dd to
>>> test block-size adjustments.
>> Bareos is not Bacula
>>
>> (well, it _IS_ Bacula, which someone has tried to repackage and claim
>> credit for. It is not supported by the Bacula community - and given the
>> legal antics of the Bareos maintainer, (including egrarious GPL
>> breaches) I would advise you don't bring Bareos problems to this forum -
>> and further suggest that you should consider uninstalling Bareos.)
>>
>>
>> Back on the block size topic:
>>
>>
>> The maximum block size supported by bacula-SD is 2MB. That is sufficient
>> to get at least 140MB/s write speed on a LTO6 drive for non-compressible
>> data and upwards of 300MB/s for highly compressible data.
>>
>> device {
>> ...
>> Maximum block size = 2M
>> ...
>> }
>>
>>
>> Attempting to increase max block size past 2MB will result in errors.
>> There is no general benefit in doing so in any case (Tested and
>> verified. I feel that setting larger block sizes should be allowable
>> even if there's no benefit, as it causes apparent errors when the max
>> block size settable in Bacula is smaller than the max allowable block
>> size reported by the tape drives)
>>
>>
>> Make sure you understand the difference between the block size (writing
>> to the tape device) and the network buffer size (communications between
>> -df, -dir and -sd - and needs to be set in all three config files if you
>> change anything)
>>
>> device {
>> ...
>> Maximum Network Buffer Size = 262144
>> ...
>> }
>>
>> Increasing the Network buffer size from the default 64kB is advisable to
>> achieve higher transfer rates on Gb/s or faster networks but there is no
>> discernable benefit going past 256kB - even on 10Gb/s networks.
>>
>>
>> You cannot change maximum block sizes on a single volume (Tape). Doing
>> so is extremely likely to result in an unreadable volume past the point
>> where the block size has changed (reading older tapes with smaller
>> maximum block sizes is still ok)
>>
>>
>> If changing block size in a working system, mark ALL open volumes as
>> used _before_ attempting any more writes.
>>
>>
>> Alan
>>
>>
>>
>>> Each test I perform I rewind the tape, write EOF and rewind again. I'm
>>> not missing a step here, right? Should I be able to write to a tape in
>>> this way with a different block size if I've used it at a different
>>> (smaller) size before?
>>>
>>> I am querying the tape-drive in the autoloader directly, the autoloader
>>> should not be part of the problem.
>>>
>>> Writing at anything under and at 256Kb works fine but is slow.
>>>
>>> The output from tapeinfo is:
>>>
>>> tapeinfo -f /dev/nst0
>>> Product Type: Tape Drive
>>> Vendor ID: 'IBM '
>>> Product ID: 'ULTRIUM-HH6 '
>>> Revision: 'G9P1'
>>> Attached Changer API: No
>>> SerialNumber: '10WT077984'
>>> MinBlock: 1
>>> MaxBlock: 8388608
>>> SCSI ID: 0
>>> SCSI LUN: 0
>>> Ready: yes
>>> BufferedMode: yes
>>> Medium Type: 0x68
>>> Density Code: 0x5a
>>> BlockSize: 0
>>> DataCompEnabled: no
>>> DataCompCapable: yes
>>> DataDeCompEnabled: yes
>>> CompType:
>>> DeCompType: 0xff
>>> Block Position: 5
>>> Partition 0 Remaining Kbytes: -1
>>> Partition 0 Size in Kbytes: -1
>>> ActivePartition: 0
>>> EarlyWarningSize: 0
>>> NumPartitions: 0
>>> MaxPartitions: 3
>>>
>>> As you can see the block size limit for the drive itself is ~8MB...
>>>
>>> Here is an output from mt:
>>>
>>> mt -f /dev/nst0 status
>>> SCSI 2 tape drive:
>>> File number=0, block number=0, partition=0.
>>> Tape block size 0 bytes. Density code 0x5a (no translation).
>>> Soft error count since last status=0
>>> General status bits on (41010000):
>>> BOT ONLINE IM_REP_EN
>>>
>>> Here is an attempt to write at a 256K block:
>>>
>>> dd if=/dev/zero of=/dev/nst0 bs=256k count=1
>>> 1+0 records in
>>> 1+0 records out
>>> 262144 bytes (262 kB) copied, 1.9402 s, 135 kB/s
>>>
>>> Here is the failure at 512K:
>>>
>>> dd if=/dev/zero of=/dev/nst0 bs=512k count=1
>>> dd: error writing ‘/dev/nst0’: Device or resource busy
>>> 1+0 records in
>>> 0+0 records out
>>> 0 bytes (0 B) copied, 1.56954 s, 0.0 kB/s
>>>
>>>
>>> It will fail with anything over 256K, even 257K.
>>>
>>> There are no errors in my system logs.
>>>
>>> I suspect either I have a configuration error here or am missing
>>> something simple OR the Adaptec 78165 raid controller is limiting the
>>> block size before I write to tape. Adaptec support is unable to confirm
>>> this. Is there a way I can prove this or does anyone have any guidance
>>> on how to continue troubleshooting this issue?
>>>
>>> Thanks!
>>>
>>> --drewv
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>>
>>> Developer Access Program for Intel Xeon Phi Processors
>>> Access to Intel Xeon Phi processor-based developer platforms.
>>> With one year of Intel Parallel Studio XE.
>>> Training and support from Colfax.
>>> Order your platform today.http://sdm.link/intel
>>> _______________________________________________
>>> Bacula-users mailing list
>>> Bacula-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>>
>>>
>
------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
