Re: [Bacula-users] file listing?
On 3/15/2011 12:41 PM, Kjetil Torgrim Homme wrote: > I wrote a script to do this for me, I call it bacula-du, since it > accepts many of the same options as du(1), and the output is the same. > > Usage: bacula-du [OPTIONS] -j JOBID > Summarize disk usage of directories included in the backup JOBID > > Options are: >-a, --all write counts for all files, not just directories >-b, --bytes use size in octets rather than number of blocks >-B, --block-size=SIZE report SIZE-byte blocks (default 1Ki) >-mlike --block-size=1Mi >-S, --separate-dirs do not include size of subdirectories >-t, --threshold=SIZE skip output for files or directories with usage > below SIZE >-L, --largest=NUM only print NUM largest directories/files > > SIZE may be (or may be an integer optionally followed by) one of following: > k (1000), Ki (1024), M (1000*1000), Mi (1024*1024), G, Gi, T, Ti, P, Pi. > > I hope others can find it useful. Can you give us an example of the output please? -- Dan Langille - http://langille.org/ -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
On 03/15/11 12:41, Kjetil Torgrim Homme wrote: > I wrote a script to do this for me, I call it bacula-du, since it > accepts many of the same options as du(1), and the output is the same. > > Usage: bacula-du [OPTIONS] -j JOBID > Summarize disk usage of directories included in the backup JOBID > > Options are: > -a, --all write counts for all files, not just directories > -b, --bytes use size in octets rather than number of blocks > -B, --block-size=SIZE report SIZE-byte blocks (default 1Ki) > -mlike --block-size=1Mi > -S, --separate-dirs do not include size of subdirectories > -t, --threshold=SIZE skip output for files or directories with usage > below SIZE > -L, --largest=NUM only print NUM largest directories/files > > SIZE may be (or may be an integer optionally followed by) one of following: > k (1000), Ki (1024), M (1000*1000), Mi (1024*1024), G, Gi, T, Ti, P, Pi. > > I hope others can find it useful. Kjetil, Thanks, this looks like a useful tool. I'm forwarding your post to the bacula-devel list to make sure folks there see it too. -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, SQL wrangler, Free Stater It's not the years, it's the mileage. -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
I wrote a script to do this for me, I call it bacula-du, since it
accepts many of the same options as du(1), and the output is the same.
Usage: bacula-du [OPTIONS] -j JOBID
Summarize disk usage of directories included in the backup JOBID
Options are:
-a, --all write counts for all files, not just directories
-b, --bytes use size in octets rather than number of blocks
-B, --block-size=SIZE report SIZE-byte blocks (default 1Ki)
-mlike --block-size=1Mi
-S, --separate-dirs do not include size of subdirectories
-t, --threshold=SIZE skip output for files or directories with usage
below SIZE
-L, --largest=NUM only print NUM largest directories/files
SIZE may be (or may be an integer optionally followed by) one of following:
k (1000), Ki (1024), M (1000*1000), Mi (1024*1024), G, Gi, T, Ti, P, Pi.
I hope others can find it useful.
--
Kjetil T. Homme
Redpill Linpro AS - Changing the game
#! /usr/bin/perl -w
# bacula-du 1.0
# Written by Kjetil Torgrim Homme
# Released under GPLv3 or the same terms as Bacula itself
sub usage {
print <<"_END_";
Usage: $0 [OPTIONS] -j JOBID
Summarize disk usage of directories included in the backup JOBID
Options are:
-a, --all write counts for all files, not just directories
-b, --bytes use size in octets rather than number of blocks
-B, --block-size=SIZE report SIZE-byte blocks (default 1Ki)
-mlike --block-size=1Mi
-S, --separate-dirs do not include size of subdirectories
-t, --threshold=SIZE skip output for files or directories with usage
below SIZE
-L, --largest=NUM only print NUM largest directories/files
SIZE may be (or may be an integer optionally followed by) one of following:
k (1000), Ki (1024), M (1000*1000), Mi (1024*1024), G, Gi, T, Ti, P, Pi.
_END_
exit(64);
}
use strict;
use DBD::mysql;
use DBI;
use MIME::Base64;
use Getopt::Long qw(:config bundling no_ignore_case);
use Data::Dumper;
my $dbhost = "localhost";
my $db = "bacula";
my $dsn = "DBI:Pg:dbname=$db;host=$dbhost";
my $dbuser = "postgres";
my $dbpass = "";
# Suggestion for MySQL:
# my $dsn = "DBI:mysql:database=mysql;mysql_read_default_group=clientp";
# my $dbuser = "mysql";
# my $dbpass = undef;
###
my $i = 0;
my %base64 = map { $_ => $i++ } split("",
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/");
sub decode_bacula_base64 {
my $acc = 0;
for (split("", $_[0])) {
$acc <<= 6;
$acc += $base64{$_};
}
return $acc;
}
sub extract_size_from_lstat {
return decode_bacula_base64((split(" ", shift))[7]);
}
sub extract_blocks_from_lstat {
return 512 * decode_bacula_base64((split(" ", shift))[9]);
}
sub convert_units {
my $num = shift;
my %units = ("k" => 1000**1, "Ki" => 1024**1, "ki" => 1024**1,
"M" => 1000**2, "Mi" => 1024**2,
"G" => 1000**3, "Gi" => 1024**3,
"T" => 1000**4, "Ti" => 1024**4,
"P" => 1000**5, "Pi" => 1024**5);
if ($num =~ /^(\d*)([kKMGTP]i?)B?$/) {
$num = ($1 ? $1 : 1) * $units{$2};
} elsif ($num !~ /^\d+$/) {
die "Can't parse: $num\n";
}
return $num;
}
### main program resumes
my $threshold = 1; # omit 0 octet sized files/directories by default
my $blocksize = 1024;
my ($jobid, $all, $bytes, $separate_dirs, $largest);
GetOptions("jobid|j=i" => \$jobid,
"threshold|t=s" => \$threshold,
"separate-dirs|S" => \$separate_dirs,
"all|a" => \$all,
"bytes|b" => \$bytes,
"block-size|B=s" => \$blocksize,
"largest|L=i" => \$largest,
"m" => sub { $blocksize = "1Mi" },
) || usage();
usage() unless $jobid;
$threshold = convert_units($threshold);
$blocksize = convert_units($blocksize);
my @padding = ("", "A==", "==", "=");
sub extract_size_from_lstat_foo {
my ($b64) = (split(" ", shift))[7];
my $acc = 0;
for (split("", decode_base64($b64 . $padding[length($b64) % 4]))) {
$acc <<= 8;
$acc += ord($_);
}
return $acc;
}
my $extract_size = $bytes
? \&extract_size_from_lstat
: \&extract_blocks_from_lstat;
my $dbh;
unless ($dbh = DBI->connect($dsn, $dbuser, $dbpass, {AutoCommit => 0})) {
print STDERR "Could not connect to database $db on host $dbhost\n";
exit 2;
}
my $sth = $dbh->prepare("
SELECT p.Path, fn.Name, LStat
FROM Path p
JOIN File f ON f.PathId = p.PathId
JOIN Filename fn ON f.FilenameId = fn.FilenameId
WHERE f.JobId = $jobid");
$sth->execute();
my %du;
my $rowcount = 0;
while (my ($path, $fname, $lstat) = $sth->fetchrow_array) {
my $size = $extract_size->($lstat);
# print STDERR "Got '$path' size $size\n";
$du{"$path$fname"} += $size if $all;
$du{$path} += $size;
next if $separate_dirs;
while ($path ne '/') {
$path =~ s,[^/]+/$,,;
$du{$pat
Re: [Bacula-users] file listing?
Hi,
Thank you for feedback again.
Yes, the BCMath module is not in default configuration if you will
compile PHP. In case compilation from sources need to add
"--enable-bcmath" to configuration parameters.
On some GNU/Linux distributions the BCMath is as default in PHP
(Debian, Slackware) and on some distributions the BCMath need to
install as a module from package apart from the PHP package (for
example Fedora).
Regards.
gani
W dniu 13 marca 2011 21:07 użytkownik Dan Langille napisał:
> On 3/12/2011 9:33 PM, ganiuszka wrote:
>>
>> W dniu 13 marca 2011 01:01 użytkownik Dan Langille
>> napisał:
>>>
>>> On 3/12/2011 9:29 AM, ganiuszka wrote:
W dniu 11 marca 2011 21:45 użytkownik ganiuszka
napisał:
>
> Hi,
> In my example jobid is putting in:
> ... File.JobId=8...
> but this "eight":
>
> base64_decode_lstat(8,File.LStat)
>
> is exactly eight field (encoded filesize field). I seem that you used
> standard base64 decoder to decode eighth field. Am I right?
>
> Bacula lstat is encoded by using non-standard base64 algorithm. Your
> decoded lstat shows like this:
>
> [dev] => 89
> [inodes] => 1366434
> [mode] => 33200
> [links] => 1
> [uid] => 80
> [gid] => 91
> [rdev] => 5481898
> [size] => 315
> [blksize] => 16384
> [blocks] => 4
> [atime] => 1299770037
> [mtime] => 1299770037
> [ctime] => 1299770066
> [LinkFl] => 0
> [flags] => 0
> [data] => 2
>
> I do not understand every field, but most fields is clear. Now, I
> wrote implementation of Bacula base64_decoder in PHP. I need finish it
> and make WebGUI for this and I will share this decoder here. For this
> I am using source code of base64 implementation in Bacula and this:
>
> http://old.nabble.com/The-File.LStat-field-td940366.html
>
> Regards.
> gani
>
> 2011/3/11 Dan Langille:
>>
>> On 3/9/2011 5:36 PM, ganiuszka wrote:
>>
>>> 2011/3/9 Mike Eggleston:
Afternoon,
I just noticed one of my clients had a huge incremental (level 2)
backup. I want to see what file caused the huge increase. I tried
'list
files jobid=20097' and though I'm shown the files, I'm not shown the
size of each file. Is there a command or query that shows me the
size
of the file?
Mike
>>
>>> Hi,
>>> Here you have SQL function for PostgreSQL (I took it from bweb):
>>>
>>> CREATE PROCEDURAL LANGUAGE plpgsql;
>>> BEGIN;
>>>
>>> CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS
>>> int8
>>> AS $$
>>> DECLARE
>>> val int8;
>>> b64 varchar(64);
>>> size varchar(64);
>>> i int;
>>> BEGIN
>>> size := split_part($2, ' ', $1);
>>> b64 :=
>>> 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
>>> val := 0;
>>> FOR i IN 1..length(size) LOOP
>>> val := val + (strpos(b64, substr(size, i, 1))-1) *
>>> (64^(length(size)-i));
>>> END LOOP;
>>> RETURN val;
>>> END;
>>> $$ language 'plpgsql';
>>>
>>> COMMIT;
>>>
>>> and this is a query which lists file path, filename and size in Bytes
>>> (in this example for jobid=8):
>>>
>>> SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS
>>> size FROM Filename, File, Path WHERE File.JobId=8 AND
>>> File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER
>>> BY size ASC;
>>>
>>> I tested it. It works.
>>
>> For the record:
>>
>> In this query, I see base64_decode_lstat(8,File.LStat)
>>
>> What is 8? It is not jobid. It is a field id. This lstat value:
>>
>> BZ FNmi IGw B BQ Bb U6Wq E7 EAA E BNeOq1 BNeOq1 BNeOrS A A C
>>
>> The eight field, as defined by ' ', is E7.
>>
>> --
>> Dan Langille - http://langille.org/
>>
>>>
>>>
Hi,
I wrote the Bacula LStat decoder implementation in PHP. I made WebGUI
for this also. Now online LStat decoder (beta version) is available on
my page Bacula.pl:
http://www.bacula.pl/bacula-lstat-decoder/
In the near future I need to make description of fields.
Here is Bacula LStat decoder function in PHP:
function decode_bacula_lstat($lstat) {
$base64 =
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
$lstat = trim($lstat);
$lstat_fields = explode(' ', $lstat);
if(count($lstat_fields) !== 16) {
die('Error! Number of lstat fields is invalid. Please
make
sure that
it is valid lstat string.');
}
list($dev, $inode, $mode, $links, $uid, $gid, $rdev, $size,
$blksize,
$blocks, $atime, $mtime, $c
Re: [Bacula-users] file listing?
On 3/12/2011 9:33 PM, ganiuszka wrote:
> W dniu 13 marca 2011 01:01 użytkownik Dan Langille
> napisał:
>> On 3/12/2011 9:29 AM, ganiuszka wrote:
>>>
>>> W dniu 11 marca 2011 21:45 użytkownik ganiuszka
>>> napisał:
Hi,
In my example jobid is putting in:
... File.JobId=8...
but this "eight":
base64_decode_lstat(8,File.LStat)
is exactly eight field (encoded filesize field). I seem that you used
standard base64 decoder to decode eighth field. Am I right?
Bacula lstat is encoded by using non-standard base64 algorithm. Your
decoded lstat shows like this:
[dev] =>89
[inodes] =>1366434
[mode] =>33200
[links] =>1
[uid] =>80
[gid] =>91
[rdev] =>5481898
[size] =>315
[blksize] =>16384
[blocks] =>4
[atime] =>1299770037
[mtime] =>1299770037
[ctime] =>1299770066
[LinkFl] =>0
[flags] =>0
[data] =>2
I do not understand every field, but most fields is clear. Now, I
wrote implementation of Bacula base64_decoder in PHP. I need finish it
and make WebGUI for this and I will share this decoder here. For this
I am using source code of base64 implementation in Bacula and this:
http://old.nabble.com/The-File.LStat-field-td940366.html
Regards.
gani
2011/3/11 Dan Langille:
>
> On 3/9/2011 5:36 PM, ganiuszka wrote:
>
>> 2011/3/9 Mike Eggleston:
>>>
>>> Afternoon,
>>>
>>> I just noticed one of my clients had a huge incremental (level 2)
>>> backup. I want to see what file caused the huge increase. I tried
>>> 'list
>>> files jobid=20097' and though I'm shown the files, I'm not shown the
>>> size of each file. Is there a command or query that shows me the size
>>> of the file?
>>>
>>> Mike
>
>> Hi,
>> Here you have SQL function for PostgreSQL (I took it from bweb):
>>
>> CREATE PROCEDURAL LANGUAGE plpgsql;
>> BEGIN;
>>
>> CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS
>> int8
>> AS $$
>> DECLARE
>> val int8;
>> b64 varchar(64);
>> size varchar(64);
>> i int;
>> BEGIN
>> size := split_part($2, ' ', $1);
>> b64 :=
>> 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
>> val := 0;
>> FOR i IN 1..length(size) LOOP
>> val := val + (strpos(b64, substr(size, i, 1))-1) *
>> (64^(length(size)-i));
>> END LOOP;
>> RETURN val;
>> END;
>> $$ language 'plpgsql';
>>
>> COMMIT;
>>
>> and this is a query which lists file path, filename and size in Bytes
>> (in this example for jobid=8):
>>
>> SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS
>> size FROM Filename, File, Path WHERE File.JobId=8 AND
>> File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER
>> BY size ASC;
>>
>> I tested it. It works.
>
> For the record:
>
> In this query, I see base64_decode_lstat(8,File.LStat)
>
> What is 8? It is not jobid. It is a field id. This lstat value:
>
> BZ FNmi IGw B BQ Bb U6Wq E7 EAA E BNeOq1 BNeOq1 BNeOrS A A C
>
> The eight field, as defined by ' ', is E7.
>
> --
> Dan Langille - http://langille.org/
>
>>>
>>>
>>>
>>
>>
>>> Hi,
>>>
>>> I wrote the Bacula LStat decoder implementation in PHP. I made WebGUI
>>> for this also. Now online LStat decoder (beta version) is available on
>>> my page Bacula.pl:
>>>
>>> http://www.bacula.pl/bacula-lstat-decoder/
>>>
>>> In the near future I need to make description of fields.
>>>
>>> Here is Bacula LStat decoder function in PHP:
>>>
>>> function decode_bacula_lstat($lstat) {
>>> $base64 =
>>> 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
>>> $lstat = trim($lstat);
>>> $lstat_fields = explode(' ', $lstat);
>>>
>>> if(count($lstat_fields) !== 16) {
>>> die('Error! Number of lstat fields is invalid. Please make
>>> sure that
>>> it is valid lstat string.');
>>> }
>>>
>>> list($dev, $inode, $mode, $links, $uid, $gid, $rdev, $size,
>>> $blksize,
>>> $blocks, $atime, $mtime, $ctime, $linkfi, $flags, $data) =
>>> $lstat_fields;
>>> $encoded_values = array('dev' => $dev, 'inode' => $inode, 'mode'
>>> =>
>>> $mode, 'links' => $links, 'uid' => $uid, 'gid' => $gid, 'rdev' =>
>>> $rdev, 'size' => $size, 'blksize' => $blksize, 'blocks' => $blocks,
>>> 'atime' => $atime, 'mtime' => $mtime, 'ctime' => $ctime, 'linkfi' =>
>>> $linkfi, 'flags' => $flags, 'data' => $data);
>>>
>>> $ret = array();
>>> foreach($encoded_values as $key => $val) {
>>> $result = 0;
>>> $is_minus = false;
>>> $start = 0;
>>>
>>> if(substr($val, 0, 1) === '-'
Re: [Bacula-users] file listing?
Hi,
First, many thanks for your feedback.
I researched that behaviour with filesize of your lstat field (
-1443891641). Yes, I made mistake with using shift bitwise operator.
>From PHP documentation:
"Warning!
Don't right shift for more than 32 bits on 32 bits systems. Don't left
shift in case it results to number longer than 32 bits."
(source: http://php.net/manual/en/language.operators.bitwise.php)
Repairing this bug is replacing the line:
$result <<= 6;
to this line:
$result = bcmul($result, bcpow(2,6));
It should work properly.
Thanks.
gani
W dniu 13 marca 2011 01:01 użytkownik Dan Langille napisał:
> On 3/12/2011 9:29 AM, ganiuszka wrote:
>>
>> W dniu 11 marca 2011 21:45 użytkownik ganiuszka
>> napisał:
>>>
>>> Hi,
>>> In my example jobid is putting in:
>>> ... File.JobId=8...
>>> but this "eight":
>>>
>>> base64_decode_lstat(8,File.LStat)
>>>
>>> is exactly eight field (encoded filesize field). I seem that you used
>>> standard base64 decoder to decode eighth field. Am I right?
>>>
>>> Bacula lstat is encoded by using non-standard base64 algorithm. Your
>>> decoded lstat shows like this:
>>>
>>> [dev] => 89
>>> [inodes] => 1366434
>>> [mode] => 33200
>>> [links] => 1
>>> [uid] => 80
>>> [gid] => 91
>>> [rdev] => 5481898
>>> [size] => 315
>>> [blksize] => 16384
>>> [blocks] => 4
>>> [atime] => 1299770037
>>> [mtime] => 1299770037
>>> [ctime] => 1299770066
>>> [LinkFl] => 0
>>> [flags] => 0
>>> [data] => 2
>>>
>>> I do not understand every field, but most fields is clear. Now, I
>>> wrote implementation of Bacula base64_decoder in PHP. I need finish it
>>> and make WebGUI for this and I will share this decoder here. For this
>>> I am using source code of base64 implementation in Bacula and this:
>>>
>>> http://old.nabble.com/The-File.LStat-field-td940366.html
>>>
>>> Regards.
>>> gani
>>>
>>> 2011/3/11 Dan Langille:
On 3/9/2011 5:36 PM, ganiuszka wrote:
> 2011/3/9 Mike Eggleston:
>>
>> Afternoon,
>>
>> I just noticed one of my clients had a huge incremental (level 2)
>> backup. I want to see what file caused the huge increase. I tried
>> 'list
>> files jobid=20097' and though I'm shown the files, I'm not shown the
>> size of each file. Is there a command or query that shows me the size
>> of the file?
>>
>> Mike
> Hi,
> Here you have SQL function for PostgreSQL (I took it from bweb):
>
> CREATE PROCEDURAL LANGUAGE plpgsql;
> BEGIN;
>
> CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS
> int8
> AS $$
> DECLARE
> val int8;
> b64 varchar(64);
> size varchar(64);
> i int;
> BEGIN
> size := split_part($2, ' ', $1);
> b64 :=
> 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
> val := 0;
> FOR i IN 1..length(size) LOOP
> val := val + (strpos(b64, substr(size, i, 1))-1) *
> (64^(length(size)-i));
> END LOOP;
> RETURN val;
> END;
> $$ language 'plpgsql';
>
> COMMIT;
>
> and this is a query which lists file path, filename and size in Bytes
> (in this example for jobid=8):
>
> SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS
> size FROM Filename, File, Path WHERE File.JobId=8 AND
> File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER
> BY size ASC;
>
> I tested it. It works.
For the record:
In this query, I see base64_decode_lstat(8,File.LStat)
What is 8? It is not jobid. It is a field id. This lstat value:
BZ FNmi IGw B BQ Bb U6Wq E7 EAA E BNeOq1 BNeOq1 BNeOrS A A C
The eight field, as defined by ' ', is E7.
--
Dan Langille - http://langille.org/
>>
>>
>>
>
>
>> Hi,
>>
>> I wrote the Bacula LStat decoder implementation in PHP. I made WebGUI
>> for this also. Now online LStat decoder (beta version) is available on
>> my page Bacula.pl:
>>
>> http://www.bacula.pl/bacula-lstat-decoder/
>>
>> In the near future I need to make description of fields.
>>
>> Here is Bacula LStat decoder function in PHP:
>>
>> function decode_bacula_lstat($lstat) {
>> $base64 =
>> 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
>> $lstat = trim($lstat);
>> $lstat_fields = explode(' ', $lstat);
>>
>> if(count($lstat_fields) !== 16) {
>> die('Error! Number of lstat fields is invalid. Please make
>> sure that
>> it is valid lstat string.');
>> }
>>
>> list($dev, $inode, $mode, $links, $uid, $gid, $rdev, $size,
>> $blksize,
>> $blocks, $atime, $mtime, $ctime, $linkfi, $flags, $data) =
>> $lstat_fields;
>> $encoded_values = array('dev' => $dev, 'inode' => $inode, 'mode' =>
>> $mode, 'links' => $links, 'uid' => $uid, 'gid' => $gid, 'rdev' =>
>> $rdev, 'size' => $size, 'blksize' => $blksize, 'blocks' => $blocks,
>> 'atime' => $atime, 'mtime' => $mtime, 'ctime' => $ctime,
Re: [Bacula-users] file listing?
On 3/12/2011 9:29 AM, ganiuszka wrote:
> W dniu 11 marca 2011 21:45 użytkownik ganiuszka napisał:
>> Hi,
>> In my example jobid is putting in:
>> ... File.JobId=8...
>> but this "eight":
>>
>> base64_decode_lstat(8,File.LStat)
>>
>> is exactly eight field (encoded filesize field). I seem that you used
>> standard base64 decoder to decode eighth field. Am I right?
>>
>> Bacula lstat is encoded by using non-standard base64 algorithm. Your
>> decoded lstat shows like this:
>>
>> [dev] => 89
>> [inodes] => 1366434
>> [mode] => 33200
>> [links] => 1
>> [uid] => 80
>> [gid] => 91
>> [rdev] => 5481898
>> [size] => 315
>> [blksize] => 16384
>> [blocks] => 4
>> [atime] => 1299770037
>> [mtime] => 1299770037
>> [ctime] => 1299770066
>> [LinkFl] => 0
>> [flags] => 0
>> [data] => 2
>>
>> I do not understand every field, but most fields is clear. Now, I
>> wrote implementation of Bacula base64_decoder in PHP. I need finish it
>> and make WebGUI for this and I will share this decoder here. For this
>> I am using source code of base64 implementation in Bacula and this:
>>
>> http://old.nabble.com/The-File.LStat-field-td940366.html
>>
>> Regards.
>> gani
>>
>> 2011/3/11 Dan Langille:
>>> On 3/9/2011 5:36 PM, ganiuszka wrote:
>>>
2011/3/9 Mike Eggleston:
>
> Afternoon,
>
> I just noticed one of my clients had a huge incremental (level 2)
> backup. I want to see what file caused the huge increase. I tried 'list
> files jobid=20097' and though I'm shown the files, I'm not shown the
> size of each file. Is there a command or query that shows me the size
> of the file?
>
> Mike
>>>
Hi,
Here you have SQL function for PostgreSQL (I took it from bweb):
CREATE PROCEDURAL LANGUAGE plpgsql;
BEGIN;
CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS int8
AS $$
DECLARE
val int8;
b64 varchar(64);
size varchar(64);
i int;
BEGIN
size := split_part($2, ' ', $1);
b64 := 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
val := 0;
FOR i IN 1..length(size) LOOP
val := val + (strpos(b64, substr(size, i, 1))-1) * (64^(length(size)-i));
END LOOP;
RETURN val;
END;
$$ language 'plpgsql';
COMMIT;
and this is a query which lists file path, filename and size in Bytes
(in this example for jobid=8):
SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS
size FROM Filename, File, Path WHERE File.JobId=8 AND
File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER
BY size ASC;
I tested it. It works.
>>>
>>> For the record:
>>>
>>> In this query, I see base64_decode_lstat(8,File.LStat)
>>>
>>> What is 8? It is not jobid. It is a field id. This lstat value:
>>>
>>> BZ FNmi IGw B BQ Bb U6Wq E7 EAA E BNeOq1 BNeOq1 BNeOrS A A C
>>>
>>> The eight field, as defined by ' ', is E7.
>>>
>>> --
>>> Dan Langille - http://langille.org/
>>>
>
>
>
> Hi,
>
> I wrote the Bacula LStat decoder implementation in PHP. I made WebGUI
> for this also. Now online LStat decoder (beta version) is available on
> my page Bacula.pl:
>
> http://www.bacula.pl/bacula-lstat-decoder/
>
> In the near future I need to make description of fields.
>
> Here is Bacula LStat decoder function in PHP:
>
> function decode_bacula_lstat($lstat) {
> $base64 =
> 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
> $lstat = trim($lstat);
> $lstat_fields = explode(' ', $lstat);
>
> if(count($lstat_fields) !== 16) {
> die('Error! Number of lstat fields is invalid. Please make sure
> that
> it is valid lstat string.');
> }
>
> list($dev, $inode, $mode, $links, $uid, $gid, $rdev, $size, $blksize,
> $blocks, $atime, $mtime, $ctime, $linkfi, $flags, $data) =
> $lstat_fields;
> $encoded_values = array('dev' => $dev, 'inode' => $inode, 'mode' =>
> $mode, 'links' => $links, 'uid' => $uid, 'gid' => $gid, 'rdev' =>
> $rdev, 'size' => $size, 'blksize' => $blksize, 'blocks' => $blocks,
> 'atime' => $atime, 'mtime' => $mtime, 'ctime' => $ctime, 'linkfi' =>
> $linkfi, 'flags' => $flags, 'data' => $data);
>
> $ret = array();
> foreach($encoded_values as $key => $val) {
> $result = 0;
> $is_minus = false;
> $start = 0;
>
> if(substr($val, 0, 1) === '-') {
> $is_minus = true;
> $start++;
> }
>
> for($i = $start; $i < strlen($val); $i++) {
> $result <<= 6;
> $result += strpos($base64, substr($val, $i , 1));
> }
> $ret[$key] = ($is_minus === true) ? -$result : $result;
> }
> return $ret;
> }
>
> and here is usage for this:
>
> $lstat = 'gB DL+b IGg B A y A D5dZR BAA fN4 BNeR+z BNeR+7 BNeR+7 A A C';
> $decoded_lstat = decode_bacu
Re: [Bacula-users] file listing?
Hi,
I wrote the Bacula LStat decoder implementation in PHP. I made WebGUI
for this also. Now online LStat decoder (beta version) is available on
my page Bacula.pl:
http://www.bacula.pl/bacula-lstat-decoder/
In the near future I need to make description of fields.
Here is Bacula LStat decoder function in PHP:
function decode_bacula_lstat($lstat) {
$base64 =
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
$lstat = trim($lstat);
$lstat_fields = explode(' ', $lstat);
if(count($lstat_fields) !== 16) {
die('Error! Number of lstat fields is invalid. Please make sure
that
it is valid lstat string.');
}
list($dev, $inode, $mode, $links, $uid, $gid, $rdev, $size, $blksize,
$blocks, $atime, $mtime, $ctime, $linkfi, $flags, $data) =
$lstat_fields;
$encoded_values = array('dev' => $dev, 'inode' => $inode, 'mode' =>
$mode, 'links' => $links, 'uid' => $uid, 'gid' => $gid, 'rdev' =>
$rdev, 'size' => $size, 'blksize' => $blksize, 'blocks' => $blocks,
'atime' => $atime, 'mtime' => $mtime, 'ctime' => $ctime, 'linkfi' =>
$linkfi, 'flags' => $flags, 'data' => $data);
$ret = array();
foreach($encoded_values as $key => $val) {
$result = 0;
$is_minus = false;
$start = 0;
if(substr($val, 0, 1) === '-') {
$is_minus = true;
$start++;
}
for($i = $start; $i < strlen($val); $i++) {
$result <<= 6;
$result += strpos($base64, substr($val, $i , 1));
}
$ret[$key] = ($is_minus === true) ? -$result : $result;
}
return $ret;
}
and here is usage for this:
$lstat = 'gB DL+b IGg B A y A D5dZR BAA fN4 BNeR+z BNeR+7 BNeR+7 A A C';
$decoded_lstat = decode_bacula_lstat($lstat);
print_r($decoded_lstat);
Any suggestions and modifications are welcome.
Regards
gani
W dniu 11 marca 2011 21:45 użytkownik ganiuszka napisał:
> Hi,
> In my example jobid is putting in:
> ... File.JobId=8...
> but this "eight":
>
> base64_decode_lstat(8,File.LStat)
>
> is exactly eight field (encoded filesize field). I seem that you used
> standard base64 decoder to decode eighth field. Am I right?
>
> Bacula lstat is encoded by using non-standard base64 algorithm. Your
> decoded lstat shows like this:
>
> [dev] => 89
> [inodes] => 1366434
> [mode] => 33200
> [links] => 1
> [uid] => 80
> [gid] => 91
> [rdev] => 5481898
> [size] => 315
> [blksize] => 16384
> [blocks] => 4
> [atime] => 1299770037
> [mtime] => 1299770037
> [ctime] => 1299770066
> [LinkFl] => 0
> [flags] => 0
> [data] => 2
>
> I do not understand every field, but most fields is clear. Now, I
> wrote implementation of Bacula base64_decoder in PHP. I need finish it
> and make WebGUI for this and I will share this decoder here. For this
> I am using source code of base64 implementation in Bacula and this:
>
> http://old.nabble.com/The-File.LStat-field-td940366.html
>
> Regards.
> gani
>
> 2011/3/11 Dan Langille :
>> On 3/9/2011 5:36 PM, ganiuszka wrote:
>>
>>> 2011/3/9 Mike Eggleston:
Afternoon,
I just noticed one of my clients had a huge incremental (level 2)
backup. I want to see what file caused the huge increase. I tried 'list
files jobid=20097' and though I'm shown the files, I'm not shown the
size of each file. Is there a command or query that shows me the size
of the file?
Mike
>>
>>> Hi,
>>> Here you have SQL function for PostgreSQL (I took it from bweb):
>>>
>>> CREATE PROCEDURAL LANGUAGE plpgsql;
>>> BEGIN;
>>>
>>> CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS int8
>>> AS $$
>>> DECLARE
>>> val int8;
>>> b64 varchar(64);
>>> size varchar(64);
>>> i int;
>>> BEGIN
>>> size := split_part($2, ' ', $1);
>>> b64 := 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
>>> val := 0;
>>> FOR i IN 1..length(size) LOOP
>>> val := val + (strpos(b64, substr(size, i, 1))-1) * (64^(length(size)-i));
>>> END LOOP;
>>> RETURN val;
>>> END;
>>> $$ language 'plpgsql';
>>>
>>> COMMIT;
>>>
>>> and this is a query which lists file path, filename and size in Bytes
>>> (in this example for jobid=8):
>>>
>>> SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS
>>> size FROM Filename, File, Path WHERE File.JobId=8 AND
>>> File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER
>>> BY size ASC;
>>>
>>> I tested it. It works.
>>
>> For the record:
>>
>> In this query, I see base64_decode_lstat(8,File.LStat)
>>
>> What is 8? It is not jobid. It is a field id. This lstat value:
>>
>> BZ FNmi IGw B BQ Bb U6Wq E7 EAA E BNeOq1 BNeOq1 BNeOrS A A C
>>
>> The eight field, as defined by ' ', is E7.
>>
>> --
>> Dan Langille - http://langille.org/
>>
--
"Większej miłości nikt nie ma nad tę, jak gdy kto życie swoje kładzie
za przyjaciół swoich." Jezus Chrystus
Re: [Bacula-users] file listing?
Hi, In my example jobid is putting in: ... File.JobId=8... but this "eight": base64_decode_lstat(8,File.LStat) is exactly eight field (encoded filesize field). I seem that you used standard base64 decoder to decode eighth field. Am I right? Bacula lstat is encoded by using non-standard base64 algorithm. Your decoded lstat shows like this: [dev] => 89 [inodes] => 1366434 [mode] => 33200 [links] => 1 [uid] => 80 [gid] => 91 [rdev] => 5481898 [size] => 315 [blksize] => 16384 [blocks] => 4 [atime] => 1299770037 [mtime] => 1299770037 [ctime] => 1299770066 [LinkFl] => 0 [flags] => 0 [data] => 2 I do not understand every field, but most fields is clear. Now, I wrote implementation of Bacula base64_decoder in PHP. I need finish it and make WebGUI for this and I will share this decoder here. For this I am using source code of base64 implementation in Bacula and this: http://old.nabble.com/The-File.LStat-field-td940366.html Regards. gani 2011/3/11 Dan Langille : > On 3/9/2011 5:36 PM, ganiuszka wrote: > >> 2011/3/9 Mike Eggleston: >>> >>> Afternoon, >>> >>> I just noticed one of my clients had a huge incremental (level 2) >>> backup. I want to see what file caused the huge increase. I tried 'list >>> files jobid=20097' and though I'm shown the files, I'm not shown the >>> size of each file. Is there a command or query that shows me the size >>> of the file? >>> >>> Mike > >> Hi, >> Here you have SQL function for PostgreSQL (I took it from bweb): >> >> CREATE PROCEDURAL LANGUAGE plpgsql; >> BEGIN; >> >> CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS int8 >> AS $$ >> DECLARE >> val int8; >> b64 varchar(64); >> size varchar(64); >> i int; >> BEGIN >> size := split_part($2, ' ', $1); >> b64 := 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'; >> val := 0; >> FOR i IN 1..length(size) LOOP >> val := val + (strpos(b64, substr(size, i, 1))-1) * (64^(length(size)-i)); >> END LOOP; >> RETURN val; >> END; >> $$ language 'plpgsql'; >> >> COMMIT; >> >> and this is a query which lists file path, filename and size in Bytes >> (in this example for jobid=8): >> >> SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS >> size FROM Filename, File, Path WHERE File.JobId=8 AND >> File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER >> BY size ASC; >> >> I tested it. It works. > > For the record: > > In this query, I see base64_decode_lstat(8,File.LStat) > > What is 8? It is not jobid. It is a field id. This lstat value: > > BZ FNmi IGw B BQ Bb U6Wq E7 EAA E BNeOq1 BNeOq1 BNeOrS A A C > > The eight field, as defined by ' ', is E7. > > -- > Dan Langille - http://langille.org/ > -- "Większej miłości nikt nie ma nad tę, jak gdy kto życie swoje kładzie za przyjaciół swoich." Jezus Chrystus -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
On 3/9/2011 5:36 PM, ganiuszka wrote: > 2011/3/9 Mike Eggleston: >> Afternoon, >> >> I just noticed one of my clients had a huge incremental (level 2) >> backup. I want to see what file caused the huge increase. I tried 'list >> files jobid=20097' and though I'm shown the files, I'm not shown the >> size of each file. Is there a command or query that shows me the size >> of the file? >> >> Mike > Hi, > Here you have SQL function for PostgreSQL (I took it from bweb): > > CREATE PROCEDURAL LANGUAGE plpgsql; > BEGIN; > > CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS int8 AS > $$ > DECLARE > val int8; > b64 varchar(64); > size varchar(64); > i int; > BEGIN > size := split_part($2, ' ', $1); > b64 := 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'; > val := 0; > FOR i IN 1..length(size) LOOP > val := val + (strpos(b64, substr(size, i, 1))-1) * (64^(length(size)-i)); > END LOOP; > RETURN val; > END; > $$ language 'plpgsql'; > > COMMIT; > > and this is a query which lists file path, filename and size in Bytes > (in this example for jobid=8): > > SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS > size FROM Filename, File, Path WHERE File.JobId=8 AND > File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER > BY size ASC; > > I tested it. It works. For the record: In this query, I see base64_decode_lstat(8,File.LStat) What is 8? It is not jobid. It is a field id. This lstat value: BZ FNmi IGw B BQ Bb U6Wq E7 EAA E BNeOq1 BNeOq1 BNeOrS A A C The eight field, as defined by ' ', is E7. -- Dan Langille - http://langille.org/ -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
Hi, Here you have SQL function for PostgreSQL (I took it from bweb): CREATE PROCEDURAL LANGUAGE plpgsql; BEGIN; CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS int8 AS $$ DECLARE val int8; b64 varchar(64); size varchar(64); i int; BEGIN size := split_part($2, ' ', $1); b64 := 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'; val := 0; FOR i IN 1..length(size) LOOP val := val + (strpos(b64, substr(size, i, 1))-1) * (64^(length(size)-i)); END LOOP; RETURN val; END; $$ language 'plpgsql'; COMMIT; and this is a query which lists file path, filename and size in Bytes (in this example for jobid=8): SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS size FROM Filename, File, Path WHERE File.JobId=8 AND File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER BY size ASC; I tested it. It works. Regards gani 2011/3/9 Mike Eggleston : > Afternoon, > > I just noticed one of my clients had a huge incremental (level 2) > backup. I want to see what file caused the huge increase. I tried 'list > files jobid=20097' and though I'm shown the files, I'm not shown the > size of each file. Is there a command or query that shows me the size > of the file? > > Mike -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
On Wed, Mar 9, 2011 at 4:52 PM, Mike Eggleston wrote: > On Wed, 09 Mar 2011, John Drescher might have said: > >> 2011/3/9 James Woodward : >> > Hello Mike, >> > I'd have to say no based on the tables defined >> > here http://bacula.org/5.0.x-manuals/en/developers/developers/Database_Tables.html. >> > The file size does not appear to be stored in the database. I don't think >> > any size is mentioned until you get to the job level. >> > James >> > >> >> I think its in the LStat. >> >> John > > In my old version (v2.0.3) I don't see an 'lstat' command. > I meant that I believe the file size and attributes are in the database in the lstat column of the File table. I am not sure there is an easy way to list this. LStat tinyblobFile attributes in base64 encoding John -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
On Wed, 09 Mar 2011, John Drescher might have said: > 2011/3/9 James Woodward : > > Hello Mike, > > I'd have to say no based on the tables defined > > here http://bacula.org/5.0.x-manuals/en/developers/developers/Database_Tables.html. > > The file size does not appear to be stored in the database. I don't think > > any size is mentioned until you get to the job level. > > James > > > > I think its in the LStat. > > John In my old version (v2.0.3) I don't see an 'lstat' command. Mike -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
2011/3/9 James Woodward : > Hello Mike, > I'd have to say no based on the tables defined > here http://bacula.org/5.0.x-manuals/en/developers/developers/Database_Tables.html. > The file size does not appear to be stored in the database. I don't think > any size is mentioned until you get to the job level. > James > I think its in the LStat. John -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
Hello Mike, I'd have to say no based on the tables defined here http://bacula.org/5.0.x-manuals/en/developers/developers/Database_Tables.html. The file size does not appear to be stored in the database. I don't think any size is mentioned until you get to the job level. James On 2011-03-09, at 12:51 PM, Mike Eggleston wrote: > Afternoon, > > I just noticed one of my clients had a huge incremental (level 2) > backup. I want to see what file caused the huge increase. I tried 'list > files jobid=20097' and though I'm shown the files, I'm not shown the > size of each file. Is there a command or query that shows me the size > of the file? > > Mike > > -- > Colocation vs. Managed Hosting > A question and answer guide to determining the best fit > for your organization - today and in the future. > http://p.sf.net/sfu/internap-sfd2d > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users > -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] file listing?
Afternoon, I just noticed one of my clients had a huge incremental (level 2) backup. I want to see what file caused the huge increase. I tried 'list files jobid=20097' and though I'm shown the files, I'm not shown the size of each file. Is there a command or query that shows me the size of the file? Mike -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
