Re: [Bacula-users] [Bacula-devel] Idea/suggestion for dedicated disk-based sd

2010-05-31 Thread Anatoly Pugachev

Here's a quick patch to src/console/console.c
with implementation to -e command line switch to bconsole.
http://snap.co.ru/~mator/one-command-console.c.patch

On 25.05.2010 / 10:33:27 -0400, Morty Abzug wrote:
 On Tue, May 25, 2010 at 05:13:36PM +0400, Anatoly Pugachev wrote:
  On 04.05.2010 / 13:33:12 -0400, Morty Abzug wrote:
 
   (12) bconsole should support an option for execute-one-command (i.e. -e)
  
  how it is different from:
  
  echo list jobs | bconsole
  
  but I've tried to implement -e  command line switch, it takes about 30
  lines of code actually...
 
 Sounds good.  :) echo list jobs | bconsole requires more shell support
 (pipes, possibly quoting) from the caller.  For example, calling
 bconsole from within bacula-dir.conf itself requires something like:
 
 RunAfterJob  = sh -c 'echo umount | bconsole'
 
 bconsole -e simplifies that.

--

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


Re: [Bacula-users] [Bacula-devel] Idea/suggestion for dedicated disk-based sd

2010-05-16 Thread Kern Sibbald
On Tuesday 04 May 2010 10:45:16 Craig Ringer wrote:
 On 4/05/2010 11:45 AM, Morty Abzug wrote:
  file dedup (rather than block dedup) could mostly be handled at the
  catalog level with another level of indirection.  I.e. instead of a
  catalog entry containing file metadata and where the file lives on
  media, it would contain file metadata and a foreign key to a table
  that maps hashes to where the file lives on the media.

 This is really hard to do when everything is bundled into big volumes,
 and volumes are expired as units. It'd be necessary to compltely
 re-design the way storage works to do file-level deduplication, and it's
 not as easy to do (efficiently, with proper retention etc) as it first
 seems either.


Actually, this is an issue that I have thought a lot about, but not from the 
stand point of having the underlying filesystem do the deduping, rather from 
the stand point of having some form of deduplication in Bacula.  

One of the problems with doing sliding block or byte level deduplication is 
that you have to efficiently store the hashes and the parts of the files that 
have been deduped.  BackupPC handles deduping files by using hard links, but 
lots of hard links are a disaster for fsck and backup programs.  Thus some 
other method is needed.

I have been wondering if git technology cannot be put to use in defining a 
new Bacula Volume format for disks.  If anyone is familiar with or wants to 
look at how git handles storing data, I would be *very* interested in your 
views.

Best regards,

Kern

--

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


Re: [Bacula-users] [Bacula-devel] Idea/suggestion for dedicated disk-based sd

2010-05-04 Thread Craig Ringer
On 4/05/2010 11:45 AM, Morty Abzug wrote:

 file dedup (rather than block dedup) could mostly be handled at the
 catalog level with another level of indirection.  I.e. instead of a
 catalog entry containing file metadata and where the file lives on
 media, it would contain file metadata and a foreign key to a table
 that maps hashes to where the file lives on the media.

This is really hard to do when everything is bundled into big volumes, 
and volumes are expired as units. It'd be necessary to compltely 
re-design the way storage works to do file-level deduplication, and it's 
not as easy to do (efficiently, with proper retention etc) as it first 
seems either.

--
Craig Ringer

--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


Re: [Bacula-users] [Bacula-devel] Idea/suggestion for dedicated disk-based sd

2010-05-04 Thread Morty Abzug
On Tue, May 04, 2010 at 04:45:16PM +0800, Craig Ringer wrote:
 On 4/05/2010 11:45 AM, Morty Abzug wrote:

 file dedup (rather than block dedup) could mostly be handled at the
 catalog level with another level of indirection.  I.e. instead of a
 catalog entry containing file metadata and where the file lives on
 media, it would contain file metadata and a foreign key to a table
 that maps hashes to where the file lives on the media.

 This is really hard to do when everything is bundled into big
 volumes, and volumes are expired as units. It'd be necessary to
 compltely re-design the way storage works to do file-level
 deduplication, and it's not as easy to do (efficiently, with proper
 retention etc) as it first seems either.

I think you could handle at least some of the retention issues by
comparing the current job's retention to the retention of the
job/volume for the old file.  If the backup of the old file doesn't
meet the new file's retention requirements, you back it up again.

However, format/architectural issues are indeed harder to overcome.
There are a number of limitations I have encountered with bacula, and
it looks like at least some of them might be fundamental to bacula's
architecture.  My list:

(1) no way to do a simple client check (AKA amcheck -c.)

(2) no way to do a simple tape check (AKA amcheck -lt)

(3) no automatic file-level deduplication.  base jobs are only a vague
approximation

(4) no way to logically group hosts into a run, with common options, the
way amanda can.  Somewhat solvable with scripting.

(5) way too complex

(6) no proper client/server model.  bacula clients need inbound ports

(7) no support for native backups.

(8) no built-in support for fully-automated storage of recovery data.
bacula recovery disks require a lot of manual work.

(9) no support for TOFU authentication.  Clients must be hand-configured.

(10) no support for Windows EFS

(11) no valid return codes from bconsole

(12) bconsole should support an option for execute-one-command (i.e. -e)

(13) bconsole should support an option for batch mode (i.e. -b)

(14) no support for disable job=$job until $time

(15) bconsole doesn't support control-c.

- Morty

--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


Re: [Bacula-users] [Bacula-devel] Idea/suggestion for dedicated disk-based sd

2010-05-03 Thread Morty Abzug
On Thu, Apr 08, 2010 at 08:39:04AM +0200, Kern Sibbald wrote:

 However, from what I see, this is basically similar to what BackuPC
 does.  The big problem I have with it is that it does not scale well
 to thousands of machines.

file dedup (rather than block dedup) could mostly be handled at the
catalog level with another level of indirection.  I.e. instead of a
catalog entry containing file metadata and where the file lives on
media, it would contain file metadata and a foreign key to a table
that maps hashes to where the file lives on the media.

- Morty

--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


Re: [Bacula-users] [Bacula-devel] Idea/suggestion for dedicated disk-based sd

2010-04-08 Thread Kern Sibbald
Hello,

I haven't seen the original messages, so I am not sure if I understand the 
full concept here so my remarks may not be pertinent.  

However, from what I see, this is basically similar to what BackuPC does.  The 
big problem I have with it is that it does not scale well to thousands of 
machines.

If I were thinking about changing the disk Volume format, I would start by 
looking at how git handles storing objects, and whether git can scale to 
handle a machine with 40 million file entries.

One thing is sure is that, unless some new way of implementing hardlinks is 
implemented, you will never see Bacula using hard links in the volumes. That 
is a sure way to make your machine unbootable if you scale large enough  Just 
backup enough clients with BackupPC and one day you will find that fsck no 
longer works.  I suspect that it will require only a couple hundred million 
hardlinks before a Linux machine will no longer boot.

Regards,

Kern

On Wednesday 07 April 2010 22:15:24 Phil Stracchino wrote:
 On 04/07/10 12:06, Robert LeBlanc wrote (in bacula-users):
  So still thinking about this, is there any reason to not have a
  hierarchical file structure for disk based backup rather than a
  serialized stream? Here are my thought, any comments welcome to have a
  good discussion about this.
 
  SD_Base_Dir
  +- PoolA
  +- PoolB
  +- JobID1
  +- JobID2
  +- Clientinfo.bacula (Bacula serial file that
  holds information similar to block header)
  +- Original File Structure (File structure from
  client is maintained and repeated here, allows for browsing of files
  outside of bacula)
   +- ClientFileA
   +- ClientFileA.bacula (Bacula serial file
  that holds information similar to the unix file attribute package)
   +- ClientFileB
   +- ClientFileB.bacula
   +- ClientDirA
   +- ClientDirA.bacula
 
  Although it's great to reuse code, I think something like this would
  be very benifical to disk based backups. The would help increase dedup
  rates and some file systems like btrfs and ZFS may be able to take
  advantage of linked files (there has been some discussion on the btrfs
  list about things like this). This would also allow it to reside on
  any file system as all the ACL and information is being serialized in
  separate files which keeps unique data out of the blocks of possible
  duplicated data. I think we could even reuse a lot of the
  serialization code, so it would just differ in how it writes the
  stream of data.

 After having thought about this a bit, I believe the idea has
 significant merit.  Tape and disk differ significantly enough that there
 is no conceptual reason not to have separate tape-specific and
 disk-specific SDs.  So long as the storage logically looks the same from
 the point of view of other daemons, the other daemons don't need to know
 that the underlying storage architecture is different.  Creating a
 hierarchical disk SD in this fashion that appears to the rest of Bacula
 exactly the same as the existing FD does, and yet takes advantage of the
 features offered by such an implementation, will not necessarily be a
 trivial problem.  It's a pretty major project and, if approved, wouldn't
 happen right away.

 The major problem I see at the moment, architecturally speaking, is that
 at the present time, this would break both migration and copy jobs
 between volumes on the new disk-only SD and volumes of any kind on the
 traditional SD, because Bacula does not yet support copy or migration
 between different SDs.  At this time, both source and destination
 devices are required to be on the same SD.



--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


Re: [Bacula-users] [Bacula-devel] Idea/suggestion for dedicated disk-based sd

2010-04-08 Thread Craig Ringer
Kern Sibbald wrote:
 Hello,
 
 I haven't seen the original messages, so I am not sure if I understand the 
 full concept here so my remarks may not be pertinent.  

Personally I wasn't suggesting a format change - I'm pretty happy with
the fake-tape volumes for on-disk storage (though I wish bextract and
bls didn't need a configured sd to read them).

My own suggestion (which started this thread) boils down to getting
decent concurrent writes to disk with a bacula-sd currently needs (IMO)
too much configuration of -sd and -dir and too much fiddling around.
Should a disk-aware sd provide a mechanism to open the same device
multiple times with DIFFERENT mounted volumes, auto-create disk devices
on demand, or otherwise provide an easier config-free way to
concurrently write to multiple volumes?.

If I wanted to propose another on-disk format, which I don't, I'd
propose zip64 with all metadata, checksum info etc not supported
directly by that format embedded in a custom-format .bacula file inside
the archive. That way backups could be extracted by something other than
bacula in a disaster recovery situation.

As it is, though, I'm pretty happy with the format, it's the way
bacula-sd acts as a gateway to disk storage that I think needs improvement.

BTW, When I suggested that greater write concurrency would be desirable
and should be easier, Phil Stracchino raised some concerns about
concurrent writes to a file system increasing fragmentation and hurting
overall performance. Rather than just wave my hands about that I've put
together a testing tool for concurrently writing big files from multiple
threads that optionally uses posix_fallocate to pre-allocate ahead in
chunks up to total file size. It tracks per-thread and total runtime and
time spent in write() and posix_fallocate() calls. I'm doing some tests
with it now and will post it to the bacula-devel list too. I'm planning
on testing the real effects of various levels of concurrent writes on
common file systems with it using a controlling shell script and will
report back when that eventually finishes.

--
Craig Ringer

--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


Re: [Bacula-users] [Bacula-devel] Idea/suggestion for dedicated disk-based sd

2010-04-08 Thread Phil Stracchino
On 04/08/10 03:53, Craig Ringer wrote:
 BTW, When I suggested that greater write concurrency would be desirable
 and should be easier, Phil Stracchino raised some concerns about
 concurrent writes to a file system increasing fragmentation and hurting
 overall performance. Rather than just wave my hands about that I've put
 together a testing tool for concurrently writing big files from multiple
 threads that optionally uses posix_fallocate to pre-allocate ahead in
 chunks up to total file size. It tracks per-thread and total runtime and
 time spent in write() and posix_fallocate() calls. I'm doing some tests
 with it now and will post it to the bacula-devel list too. I'm planning
 on testing the real effects of various levels of concurrent writes on
 common file systems with it using a controlling shell script and will
 report back when that eventually finishes.

I'll be interested to see those results.  Which filesystems are you testing?

-- 
  Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355
  ala...@caerllewys.net   ala...@metrocast.net   p...@co.ordinate.org
 Renaissance Man, Unix ronin, Perl hacker, Free Stater
 It's not the years, it's the mileage.

--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


Re: [Bacula-users] [Bacula-devel] Idea/suggestion for dedicated disk-based sd

2010-04-08 Thread Craig Ringer
Phil Stracchino wrote:
 On 04/08/10 03:53, Craig Ringer wrote:
 BTW, When I suggested that greater write concurrency would be desirable
 and should be easier, Phil Stracchino raised some concerns about
 concurrent writes to a file system increasing fragmentation and hurting
 overall performance. Rather than just wave my hands about that I've put
 together a testing tool for concurrently writing big files from multiple
 threads that optionally uses posix_fallocate to pre-allocate ahead in
 chunks up to total file size. It tracks per-thread and total runtime and
 time spent in write() and posix_fallocate() calls. I'm doing some tests
 with it now and will post it to the bacula-devel list too. I'm planning
 on testing the real effects of various levels of concurrent writes on
 common file systems with it using a controlling shell script and will
 report back when that eventually finishes.
 
 I'll be interested to see those results.  Which filesystems are you testing?

I'm interested in ext3, ext4 and xfs. I should probably look at zfs too,
but don't have any hosts that it runs on usefully and don't really have
any personal interest in it.

I've posted the tool's sources  makefile to the devel list along with
some preliminary notes. So far it looks like posix_fallocate is, as
expected, a win if you know the volume size (or a reasonable
approximation of it) in advance, and not much help or a loss if you
don't so you posix_fallocate(...) smaller chunks progressively. Further
info will appear on -devel once I've put it together.

--
Craig Ringer

--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


Re: [Bacula-users] [Bacula-devel] Idea/suggestion for dedicated disk-based sd

2010-04-08 Thread Robert LeBlanc
On Thu, Apr 8, 2010 at 12:39 AM, Kern Sibbald k...@sibbald.com wrote:
 Hello,

 I haven't seen the original messages, so I am not sure if I understand the
 full concept here so my remarks may not be pertinent.

 However, from what I see, this is basically similar to what BackuPC does.  The
 big problem I have with it is that it does not scale well to thousands of
 machines.

 If I were thinking about changing the disk Volume format, I would start by
 looking at how git handles storing objects, and whether git can scale to
 handle a machine with 40 million file entries.

 One thing is sure is that, unless some new way of implementing hardlinks is
 implemented, you will never see Bacula using hard links in the volumes. That
 is a sure way to make your machine unbootable if you scale large enough  Just
 backup enough clients with BackupPC and one day you will find that fsck no
 longer works.  I suspect that it will require only a couple hundred million
 hardlinks before a Linux machine will no longer boot.


It wasn't my intention that Bacula try to create the hard links like
BackupPC, I figured that if someone wanted to do that, they could run
a script outside of Bacula. I'm thinking of the ability to offload the
data compression to the file system in general, or alternatively have
Bacula compress it. The reason being is that with Bacula's current
tape format, dedup technologies can not dedup it very well. From what
I can tell of the tape format, every 64K of duplicate data had a
unique header rendering it unique and therefore not a candidate for
dedup.

I had two ideas for trying to overcome this problem, one was to have a
slightly modified Bacula tape format for disks that would move the
unique header information to the front or the back of the job stream,
and the format would create a sparse file with job files starting at a
user defined blocksize. I then thought about storing tier 3 data on
the same dedup device or file system and that if done a certain way we
could get 'free' backups. If Bacula backed-up to the same device with
a hierarchical file system approach, then the original files and the
files that Bacula backed up would look the same. Plus it would be easy
to recover in the case of total failures of Bacula-dir and sd (I'm
thinking disaster recovery).

I've been running Bacula backups on a dedup box for almost a year and
can't get better than 4x when I believe the data that we have should
be about 10x. With dedup becoming more popular, I'm just trying to
make Bacula even more appealing for those who want to dedup. If people
are using straight disk, then compression could be enabled by bacula
and the format might be a little different (like tar bz2 archives),
but most newer file systems are starting to support on the fly
compression, so I don't know how critical it is.

These are all ideas to get some discussion about how, if a file aware
SD is implemented, what may be good to offer maximum flexibility and
be able to leverage features that are being implemented in current and
future file systems.

Thanks,

Robert LeBlanc
Life Sciences  Undergraduate Education Computer Support
Brigham Young University

--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


Re: [Bacula-users] [Bacula-devel] Idea/suggestion for dedicated disk-based sd

2010-04-08 Thread Phil Stracchino
On 04/08/10 09:13, Craig Ringer wrote:
 Phil Stracchino wrote:
 I'll be interested to see those results.  Which filesystems are you testing?
 
 I'm interested in ext3, ext4 and xfs. I should probably look at zfs too,
 but don't have any hosts that it runs on usefully and don't really have
 any personal interest in it.

I have a big ZFS box I can run tests on.


-- 
  Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355
  ala...@caerllewys.net   ala...@metrocast.net   p...@co.ordinate.org
 Renaissance Man, Unix ronin, Perl hacker, Free Stater
 It's not the years, it's the mileage.

--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


Re: [Bacula-users] [Bacula-devel] Idea/suggestion for dedicated disk-based sd

2010-04-08 Thread Richard Scobie
Craig Ringer wrote:

 I'm interested in ext3, ext4 and xfs. I should probably look at zfs too,
 but don't have any hosts that it runs on usefully and don't really have
 any personal interest in it.

You may find the XFS mount directive, filestreams of benefit here. 
There is not much documentation about it, but here is some tutorial 
information:

Filestreams Allocator

A certain class of applications such as those doing film scanner
video ingest will write many large files to a directory in sequence.
It's important for playback performance that these files end up
allocated next to each other on disk, since consecutive data is
retrieved optimally by hardware RAID read-ahead.

XFS's standard allocator starts out doing the right thing as far
as file allocation is concerned.  Even if multiple streams are being
written simultaneously, their files will be placed separately and
contiguously on disk.  The problem is that once an allocation group
fills up, a new one must be chosen and there's no longer a parent
directory in a unique AG to use as an AG owner.  Without a way
to reserve the new AG for the original directory's use, all the
files being allocated by all the streams will start getting placed
in the same AGs as each other.  The result is that consecutive
frames in one directory are placed on disk with frames from other
directories interleaved between them, which is a worst-case layout
for playback performance.  When reading back the frames in directory
A, hardware RAID read-ahead will cache data from frames in directory
B which is counterproductive.

Create a file system with a small AG size to demonstrate:

sles10:~ sjv: sudo mkfs.xfs -d agsize=64m /dev/sdb7  /dev/null
sles10:~ sjv: sudo mount /dev/sdb7 /test
sles10:~ sjv: sudo chmod 777 /test
sles10:~ sjv: cd /test
sles10:/test sjv:

Create ten 10MB files concurrently in two directories:

sles10:/test sjv: mkdir a b
sles10:/test sjv: for dir in a b; do
   for file in `seq 0 9`; do
   xfs_mkfile 10m $dir/$file
   done 
   done; wait 2/dev/null
[1] 30904
[2] 30905
sles10:/test sjv: ls -lid * */*
131 drwxr-xr-x 2 sjv users   86 2006-10-20 13:48 a
132 -rw--- 1 sjv users 10485760 2006-10-20 13:48 a/0
133 -rw--- 1 sjv users 10485760 2006-10-20 13:48 a/1
134 -rw--- 1 sjv users 10485760 2006-10-20 13:48 a/2
135 -rw--- 1 sjv users 10485760 2006-10-20 13:48 a/3
136 -rw--- 1 sjv users 10485760 2006-10-20 13:48 a/4
137 -rw--- 1 sjv users 10485760 2006-10-20 13:48 a/5
138 -rw--- 1 sjv users 10485760 2006-10-20 13:48 a/6
139 -rw--- 1 sjv users 10485760 2006-10-20 13:48 a/7
140 -rw--- 1 sjv users 10485760 2006-10-20 13:48 a/8
141 -rw--- 1 sjv users 10485760 2006-10-20 13:48 a/9
262272 drwxr-xr-x 2 sjv users   86 2006-10-20 13:48 b
262273 -rw--- 1 sjv users 10485760 2006-10-20 13:48 b/0
262274 -rw--- 1 sjv users 10485760 2006-10-20 13:48 b/1
262275 -rw--- 1 sjv users 10485760 2006-10-20 13:48 b/2
262276 -rw--- 1 sjv users 10485760 2006-10-20 13:48 b/3
262277 -rw--- 1 sjv users 10485760 2006-10-20 13:48 b/4
262278 -rw--- 1 sjv users 10485760 2006-10-20 13:48 b/5
262279 -rw--- 1 sjv users 10485760 2006-10-20 13:48 b/6
262280 -rw--- 1 sjv users 10485760 2006-10-20 13:48 b/7
262281 -rw--- 1 sjv users 10485760 2006-10-20 13:48 b/8
262282 -rw--- 1 sjv users 10485760 2006-10-20 13:48 b/9
sles10:/test sjv:

Note that all the inodes are in the same AGs as each other.  What
about the file data?  Use xfs_bmap -v to examine the extents:

sles10:/test sjv: for file in `seq 0 9`; do
   bmap_a=`xfs_bmap -v a/$file | tail -1`
   bmap_b=`xfs_bmap -v b/$file | tail -1`
   ag_a=`echo $bmap_a | awk '{print $4}'`
   ag_b=`echo $bmap_b | awk '{print $4}'`
   br_a=`echo $bmap_a | awk '{printf %-18s, $3}'`
   br_b=`echo $bmap_b | awk '{printf %-18s, $3}'`
   echo a/$file: $ag_a $br_a b/$file: $ag_b $br_b
   done
a/0: 0 96..20575  b/0: 1 131168..151647
a/1: 0 20576..41055   b/1: 1 151648..172127
a/2: 0 41056..61535   b/2: 1 172128..192607
a/3: 0 61536..82015   b/3: 1 192608..213087
a/4: 0 82016..102495  b/4: 1 213088..233567
a/5: 0 102496..122975 b/5: 1 233568..254047
a/6: 2 299600..300111 b/6: 2 262208..275007
a/7: 2 338016..338527 b/7: 2 312400..312911
a/8: 2 344672..361567 b/8: 3 393280..401983
a/9: 2 361568..382047 b/9: 3 401984..421951
sles10:/test sjv:

The middle column is the AG number and the right column is the block
range.  Note how the extents for files in both directories get
placed on top of each other in AG 2.

Something to note in the results is that even though the file extents
have worked their way up into AGs 2 and 3, the inode numbers show
that the file inodes are all in the same AGs as their parent
directory, i.e. AGs 0 and 1.  Why is this?  To understand, it's
important to consider the order in which events are occurring.  The
two bash processes writing files are calling xfs_mkfile, which
starts by 

Re: [Bacula-users] [Bacula-devel] Idea/suggestion for dedicated disk-based sd

2010-04-08 Thread Gary R. Schmidt
On Fri, April 9, 2010 05:09, Richard Scobie wrote:
[snip]
 You may find the XFS mount directive, filestreams of benefit here.

And using the 64-bit XFS will also better[1] than the standard 32-bit XFS.
 Or the hybrid 32-bit XFS using 64-bit layout rules.  (Damn, I only left
SGI at the end of 2008 and I've already forgotten all the stuff I used to
know about the inner machinations of XFS!!  (I wasn't an XFS developer, I
started on CXFS and ended up in QA for the NAS/CXFS/XFS stuff))

And, in re ZFS, there is no posix_fallocate equivalent in S10 (that I
can find).  There may be one in OpenSolaris, but I dunno.

Cheers,
GaryB-)

1 - Oh, so, so, so much better than the pathetically choked whimpering
little rodent that is 32-bit XFS.  Switching to the 64-bit layout as the
default was prevented because the Linux stat system call could not
safely handle 64-bit inodes.  Lusers.


--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


Re: [Bacula-users] [Bacula-devel] Idea/suggestion for dedicated disk-based sd

2010-04-08 Thread Richard Scobie
Gary R. Schmidt wrote:

 And using the 64-bit XFS will also better[1] than the standard 32-bit XFS.

That would be using the inode64 mount option.

Regards,

Richard

--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users