Re: du(1) for FTP sites
On 05/02/2010 12:29 AM, Benjamin Scott wrote: It's still very susceptible to differences in FTP directory format. I've confirmed a Microsoft FTP server will cause it to choke horribly. Did you look at FUSE mounting the directory and running 'du' on it? -Bill -- Bill McGonigle, Owner BFC Computing, LLC http://bfccomputing.com/ Telephone: +1.603.448.4440 Email, IM, VOIP: b...@bfccomputing.com VCard: http://bfccomputing.com/vcard/bill.vcf Social networks: bill_mcgonigle/bill.mcgonigle ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: du(1) for FTP sites
On 05/01/2010 08:30 PM, Alan Johnson wrote: fuseftp mount then du? ack, IMAP just pulled in Alan's message. :P -Bill -- Bill McGonigle, Owner BFC Computing, LLC http://bfccomputing.com/ Telephone: +1.603.448.4440 Email, IM, VOIP: b...@bfccomputing.com VCard: http://bfccomputing.com/vcard/bill.vcf Social networks: bill_mcgonigle/bill.mcgonigle ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: du(1) for FTP sites
On Mon, May 3, 2010 at 2:50 PM, Bill McGonigle b...@bfccomputing.com wrote: Did you look at FUSE mounting the directory and running 'du' on it? Several people suggested that (on-list and off), and I may explore it some day, but I liked the script approach better. For one, it has at least a chance at being cross-platform on the client. Plus pretending FTP is a filesystem just rubs me the wrong way. I guess there's technically no reason why it can't be used that way, but it still gives me the heebie-jeebies. Maybe I'm just getting old... -- Ben ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: du(1) for FTP sites
On Sun, May 2, 2010 at 12:29 AM, Benjamin Scott dragonh...@gmail.com wrote: http://sites.google.com/site/mailvortex/unix-and-linux/duftp It may even be cross-platform, although I haven't tried it anywhere but *nix yet. I can confirm that it works on Windows XP with ActivePerl. Woot! -- Ben ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: du(1) for FTP sites
On 05/02/2010 12:29 AM, Benjamin Scott wrote: Kevin's wget-based implementation worked (thanks again, Kevin!), but was slow due to repeated invocations of wget. I looked at the code to see if I could do something about that... and that awoke the coding bug in me. End result is implemented in terms of Perl Net::FTP and needs no external programs. It may even be cross-platform, although I haven't tried it anywhere but *nix yet. It's still very susceptible to differences in FTP directory format. I've confirmed a Microsoft FTP server will cause it to choke horribly. :) http://sites.google.com/site/mailvortex/unix-and-linux/duftp Suggestion: change the Synopsis syntax from 'duext' to 'duftp' Perennially RTFM'ing, -- Ted Roche Ted Roche Associates, LLC http://www.tedroche.com ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: du(1) for FTP sites
On Sun, May 2, 2010 at 10:00 AM, Ted Roche tedro...@tedroche.com wrote: http://sites.google.com/site/mailvortex/unix-and-linux/duftp Suggestion: change the Synopsis syntax from 'duext' to 'duftp' Already fixed on the website and in my local copy; I just didn't bother uploading a new file for just that. There will always be at least one stupid typo you don't notice until just after you submit something. -- Ben ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: du(1) for FTP sites
I started looking at writing it in Python. The library I was looking at (ftputil http://ftputil.sschwarzer.net/ , easy_install ftputil) supports walking the directory structure (much like os.walk or is it os.path.walk) and provides a stat method to get info (including filesize) on the files (therefore should be FTP server agnostic). However it's the end of the semester, and getting school backlog done has been more important than writing a fun little Python script. :( Glad you got something working to solve it and for sharing it! -Shawn On Sun, May 2, 2010 at 12:29 AM, Benjamin Scott dragonh...@gmail.comwrote: Kevin's wget-based implementation worked (thanks again, Kevin!), but was slow due to repeated invocations of wget. I looked at the code to see if I could do something about that... and that awoke the coding bug in me. End result is implemented in terms of Perl Net::FTP and needs no external programs. It may even be cross-platform, although I haven't tried it anywhere but *nix yet. It's still very susceptible to differences in FTP directory format. I've confirmed a Microsoft FTP server will cause it to choke horribly. :) http://sites.google.com/site/mailvortex/unix-and-linux/duftp Share and enjoy! -- Ben ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: du(1) for FTP sites
On 05/02/2010 11:05 AM, Benjamin Scott wrote: There will always be at least one stupid typo you don't notice until just after you submit something. The Joy of Open Source: there'll always be one typo and twenty people to point it out to you :) -- Ted Roche Ted Roche Associates, LLC http://www.tedroche.com ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: du(1) for FTP sites
On Sun, May 2, 2010 at 12:08 PM, Shawn O'Shea sh...@eth0.net wrote: ... provides a stat method to get info (including filesize) on the files (therefore should be FTP server agnostic). The problem with FTP is that the output of LIST is implementation-specific; it varies from server to server. There's a fair bit of variation even within system types. Just because you know the system type is UNIX doesn't mean you're going to get exactly the same format as some other server. Some provide a total line; the date format varies; etc. According to the docs for ftputil, it tries to parse output for UNIX and MSFT style directory listings, which is more than my script does, but it's still vulnerable to the vagaries of output formats. (And won't work at all for a VMS server. ;-) ) RFC-3659 (2007) defines some new commands, notably MLST, which is intended to give a directory list for machine processing, in a well-defined format. I don't know how widely implemented MLST is, though. (i.e, if you do actually find a VMS FTP server, it's unlikely to be running software new enough to implement MLST.) It's unfortunate that it took to 2007 for a machine-friendly format to be defined. Of course, HTTP file servers are even worse. Sigh. -- Ben ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: du(1) for FTP sites
Benjamin Scott writes: Kevin's wget-based implementation worked (thanks again, Kevin!), but was slow due to repeated invocations of wget. Yeah, the big design principal behind my implementation was that I was trying to get it done in less than ~30 minutes...before I had to get going home. Regards, --kevin -- alumni.unh.edu!kdc / http://kdc-blog.blogspot.com/ GnuPG: D87F DAD6 0291 289C EB1E 781C 9BF8 A7D8 B280 F24E Wipe him down with gasoline 'til his arms are hard and mean From now on boys this iron boat's your home So heave away, boys. -- Tom Waits ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: du(1) for FTP sites
On Sun, May 2, 2010 at 9:39 PM, Kevin D. Clark kevin_d_cl...@comcast.net wrote: Kevin's wget-based implementation worked (thanks again, Kevin!), but was slow due to repeated invocations of wget. Yeah, the big design principal behind my implementation was that I was trying to get it done in less than ~30 minutes...before I had to get going home. Oh, I understand completely, and indeed, the available right now aspect came in very useful for me. It was only the next day that I started toying around with the code and things kind of snowballed into a multi-hour research/coding/testing session. Those feeping creatures are dangerous! ;-) -- Ben ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: du(1) for FTP sites
On Fri, Apr 30, 2010 at 6:28 PM, Kevin D. Clark kevin_d_cl...@comcast.net wrote: [attached] gives a big skeleton of what you are looking for. The code itself could definitely be improved. Wow, did you write that just for this request? Thanks! -- Ben ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: du(1) for FTP sites
On Fri, Apr 30, 2010 at 4:10 PM, Benjamin Scott dragonh...@gmail.comwrote: I'm looking for something like du(1), except taking an FTP site instead of a local directory path. Trying to scope out disk usage on an FTP site I don't have shell access to. Non-GUI strongly preferred, but I'll take what I can get. fuseftp mount then du? ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: du(1) for FTP sites
Kevin's wget-based implementation worked (thanks again, Kevin!), but was slow due to repeated invocations of wget. I looked at the code to see if I could do something about that... and that awoke the coding bug in me. End result is implemented in terms of Perl Net::FTP and needs no external programs. It may even be cross-platform, although I haven't tried it anywhere but *nix yet. It's still very susceptible to differences in FTP directory format. I've confirmed a Microsoft FTP server will cause it to choke horribly. :) http://sites.google.com/site/mailvortex/unix-and-linux/duftp Share and enjoy! -- Ben ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: du(1) for FTP sites
Benjamin Scott writes:
I'm looking for something like du(1), except taking an FTP site
instead of a local directory path. Trying to scope out disk usage on
an FTP site I don't have shell access to.
Non-GUI strongly preferred, but I'll take what I can get.
[attached] gives a big skeleton of what you are looking for. The code
itself could definitely be improved.
I tested this with two FTP servers, but I know for a fact that the
number of variations here is huge.
Enjoy!
--kevin
--
alumni.unh.edu!kdc / http://kdc-blog.blogspot.com/
GnuPG: D87F DAD6 0291 289C EB1E 781C 9BF8 A7D8 B280 F24E
Wipe him down with gasoline 'til his arms are hard and mean
From now on boys this iron boat's your home
So heave away, boys.
-- Tom Waits
#!/usr/bin/perl
# author: kevin d. clark (alumni.unh.edu!kdc)
use warnings;
use strict;
use Getopt::Long;
use IO::File;
# given a ftp URL for a directory, this function determines how much
# disk space this directory (and all sub-directories) use
sub get_disk_space($$$) {
my ($url, $user_args, $password_args) = @_;
my $total = 0;
my @dirs;
my $wget_cmd = wget $user_args $password_args --quiet -O /dev/null
--no-remove-listing --waitretry=10 --retry-connrefused '$url';
system($wget_cmd) die some problem with wget: here was the
command\n\n$wget_cmd\n\nexit value was $?\n;
my $fh = new IO::File(.listing, r)
|| die could not open '.listing': $!\n;
while (my $line = $fh-getline()) {
1 while ($line =~ s/[\012\015]$//);
my ($size, $name) = $line =~ m/\S+\s+ # permissions
\S+\s+ # can't entirely remember...
\S+\s+ # user?
\S+\s+ # group?
(\S+\s+) # size
\S+\s+ # month
\S+\s+ # day
\S+\s+ # time
(.*) # NAME
/x;
if ($line =~ /^d/ $name ne . $name ne ..) {
push @dirs, $name;
}
else {
$total += $size;
}
}
$fh-close() || die Can't close file: $!\n;
map {
my $u = $url . / . $_ . /;
$total += get_disk_space($u, $password_args, $user_args)
} @dirs;
return $total;
}
main: {
my ($user, $password);
my ($user_args, $password_args, $url) = (, , );
if (! GetOptions(user=s = \$user,
password=s = \$password,
url=s = \$url,
)) {
die(Bad command line arguments!\n);
}
die You didn't specify a URL! if ($url eq );
$user_args = --user '$user' if (defined($user));
$password_args = --password '$password' if (defined($password));
my $total = get_disk_space($url, $user_args, $password_args);
print $url\t\t$total\n;
}
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: du(1) for FTP sites
I wrote: [attached] gives a big skeleton of what you are looking for. The code itself could definitely be improved. Oh yeah, invoke it thusly: remote-du --url 'ftp://ftp.gnu.org/pub/old-gnu/Manuals/bfd-2.9.1/' --kevin -- alumni.unh.edu!kdc / http://kdc-blog.blogspot.com/ GnuPG: D87F DAD6 0291 289C EB1E 781C 9BF8 A7D8 B280 F24E Wipe him down with gasoline 'til his arms are hard and mean From now on boys this iron boat's your home So heave away, boys. -- Tom Waits ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
