Re: [OpenAFS] OpenAFS on windows - profile in AFS, who uses it?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Lars Schimmer wrote: HI! Just out of interest: which site (beside our cell) is using OpenAFS as filestore for Windows Profile and customers working on this? Good to see we are not the only cell working with profiles in AFS space. We use 1.4.6 linux servers, Win XP SP2 and at least 1.5.27 OpenAFS Windows clients. So far it works mostly. Biggest problems: 1. slowness - Users reports of LONG loading times for 400-1000 MB roaming profiles, up to 20 minutes 2. as in computergraphic, we handle 10-400 MB models with our programs. Getting this via the fileserver at 5-6 MB/sec is kinda slow. The cache on AFS is written as a single file, I would prefer a partition as it can't be defragmented 3. reliability - some days windows hit a error while saving the profile and no readable info could be found - I assume mostly unicode problems 4. the context menu - nearly all workstations has it disabled as it send the explorer into a timeout - from 1.5.x on til now for the most active win users the context menu is a problem - right click on a file OUT of AFS space keep the system freeze for ~60 sec til the context menu appears. For files in AFS space it appears direct. Til yet I haven't found any solution why it freezes, as in all my test accounts it just doesn't appear, even on workstations on which the other users have that problem 5. Office - haven't tested with latest OpenAFS, but nearly 1 year ago one workstation had big problems with office files in AFS space - read/write a file to AFS freezed the machine for 1-2 minutes, even with debugging on we could only say openAFS works correct. We only got 5-10 people working every day with windows roaming profile in AFS space, maybe this list seems to be harsh, but thats just collected issues/probs/wishes from 2 years practise. Some can't be solved, e.g. speed of network. I just want to know if these problems are problems to the others or if these are not known so far... (and yes, I know some parts are solved IF the new binding of OpenAFS as a native Windows drive is ready) MfG, Lars Schimmer - -- - - TU Graz, Institut für ComputerGraphik WissensVisualisierung Tel: +43 316 873-5405 E-Mail: [EMAIL PROTECTED] Fax: +43 316 873-5402 PGP-Key-ID: 0x4A9B1723 -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHruMbmWhuE0qbFyMRAiM/AJ0VfJn54lCk0xIxZVkNdHH27+qf2wCfVCpt 5ZEk3uyiCOOLZ37cO4CCNtI= =sJQf -END PGP SIGNATURE- ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] OpenAFS on windows - profile in AFS, who uses it?
Rodney M. Dyer [EMAIL PROTECTED] wrote: So the only variable we have left is %username%. How am I supposed to setup folder redirection? I can't use: n:\cell\usr\a\%username%\pc\win_data\Desktop That won't work since the parent folders are different for every user. I have not tested this (all my user directories are out of a single folder) but can one use the documented set command envirnoment variable display options (set /?) to obtain the first (second, third, etc.) letter of a username? C:\echo %USERNAME% Administrator C:\echo %USERNAME:~0,1% A C:\echo %USERNAME:~1,1% d CDC -- Christopher D. Clausen ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] OpenAFS on windows - profile in AFS, who uses it?
At 04:27 PM 2/10/2008, Christopher D. Clausen wrote: I have not tested this (all my user directories are out of a single folder) but can one use the documented set command envirnoment variable display options (set /?) to obtain the first (second, third, etc.) letter of a username? I haven't tested it, but I really doub't that would work since the CMD.EXE shell is not what is responsible for the redirection operation. It's only under the command shell that you have the option of performing environment variable string manipulations. Besides, if I take your meaning, you are assuming that the users account names under /afs/cell/usr/a all begin with the 'a' character in the path. At our site, they do not. The algorithm at our site was created long ago, and not by me. The algorithm we use is simply to spread the accounts accross the alphabetic characters evenly so the number of accounts under each letter is generally the same. This was to reduce the stat'ing time of the subfolders. Rodney ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] OpenAFS on windows - profile in AFS, who uses it?
At 03:39 PM 2/10/2008, Rodney M. Dyer wrote: When a user is logged on, a global drive cannot be unmounted by the user which is nice. (This assumes they aren't an administrator.) I'm replying to myself here because I forgot a couple of extra items at this point. When a global drive is mounted, and the user logs on. If you open an Explorer file browser the drive will show up as a Disconnected Network Drive. You are not able to change the label of this drive because the Explorer file browser does not consider that drive as yours, under your explorer shell mount table. This is not a problem however because the drive operates as normal. It's just ugly for the user to see the disconnected part. We get questions from our users about this from time to time. Others just don't care. Related and ugly messy stuff related to the global drive... If you have roaming profiles for your users, and you are using a global drive, then you need to figure out what to do about this registry key... [HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\MountPoints2] This is the users explorer shell mount table. Actually the disconnected drive is listed there in the mount table, but the Explorer lists it (recreates it) at every logon as a different GUID and so we can't do anything with that since we have no idea which GUID is the proper one to change (if we want to set a drive label). As the user logs on and off over time this table will normally be managed by the shell for you. However, if you are using global drives, the explorer shell will not ever remove those newly created GUIDs and it will begin to fill up over time with registry entries. This is BAD. At our site, since we have the luxury of being fully in control the users desktop experience, I decided to completely erase this folder at every user logon, using a user logon script. Our user logon script runs before the explorer shell starts, so there is no interference with explorer while performing this deletion. Since the MountPoints2 folder is the root of a tree of registry keys, you need to delete that folder and everything under it. I found the MountPoints2 key quite by accident last year when doing research on why a drive label can't be set for Disconnected Drives. It was then that I observed that it filled up over time. Once the user is logged on, if you create and other driver mounts to AFS for the user in a user logon script, then you CAN set the label name. Here's the code I use for example... Note: this is the forced way by direct registry manipulation. Microsoft frowns on this because you aren't supposed to go changing things behind the Explorers back, and besides, according to Microsoft you shouldn't change anything under their Microsoft key anyway. Drive mount in user logon script: net use u: \\afs\%username% ntregedt AddValue HKEY_CURRENT_USER Software\Microsoft\Windows\CurrentVersion\Explorer\MountPoints2\##AFS#%username% _LabelFromReg REG_SZ AFS Unix Drive Here's the more appropriate programatic way (using vb scripting): Set oShell = CreateObject(Shell.Application) mDrive = U: oShell.NameSpace(mDrive).Self.Name = AFS Unix Drive If you've understood this, then you must be sober, so enjoy, Rodney ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
RE: [OpenAFS] byte-level incremental backups using full+reverse delta
Well as we are giving away scripting for replication...
This script works *without* configuration and maybe people can learn from
it.
Just run it on volservers and don't forget to get your AFStoken first.
repl_afs.pl
--
#!/usr/bin/perl -w
#
# OpenAFS volume replication
# by : Marcel D.A. Koopmans of Elysium Open Systems
#
# version : 1.0.5 ( 2007-08-24 20:25 GMT+1 )
use strict;
use Sys::Syslog;
sub write_log {
my ( ${level}, ${message} ) = @_;
openlog( afs_repl, , DAEMON );
syslog( ${level}, ${message} .\n );
closelog;
}
sub main {
my ${err}=0;
my ( $a, $b );
my @temp;
my ( @ip_a, %ip_h );
my ( %hn_h );
my ( $volume, $sites, $host );
my ( ${exitcode} );
# log start
write_log( INFO, OpenAFS replication start );
# get local IP addresses
open ( FILE, /sbin/ifconfig -a | );
while ( FILE ) {
chomp($_);
if ( $_=~ m/^\s+inet\s+addr:([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\s+.*$/ ) {
if ( ! exists ( ${ip_h{$1}} ) ) {
${ip_h{$1}}=1;
${ip_a[$#{ip_a}+1]}=$1;
}
}
}
close ( FILE );
# sort IP addresses;
@[EMAIL PROTECTED];
@ip_a=sort(@temp);
$#temp=-1;
# resolve hostnames
if ( $#{ip_a} -1 ) {
for ( $a=0; $a=$#{ip_a}; $a++ ) {
open ( FILE, /usr/bin/getent hosts . ${ip_a[$a]} . | );
while ( FILE ) {
chomp($_);
@temp=split(/\s+/,$_);
for ( $b=0; $b=$#temp; $b++ ) {
if ( ! exists ( ${hn_h{$temp[$b]}} ) ) {
${hn_h{$temp[$b]}}=1;
}
}
}
close ( FILE );
}
}
# Get openAFS volumes
$a=-1;
open ( FILE, /usr/bin/vos listvldb 2 /dev/null | );
while ( FILE ) {
chomp($_);
$_=~ s/\s+$//;
if ( ( $a == 2 ) ( $_=~
m/^\s+server\s+(.+)\s+partition\s+\/vicep[a-z]+\s+RW\s+Site$/ ) ) {
${host}=$1;
if ( exists ( $hn_h{$host} ) ) {
if ( ${sites} 1 ) {
${exitcode}=system(/usr/bin/vos release . ${volume} .
/dev/null 21 );
${exitcode}=${exitcode} / 256;
if ( ${exitcode} == 0 ) {
write_log( INFO, release of volume . ${volume} .
success );
} else {
write_log( ERR, release of volume . ${volume} . failed,
returned exit code . ${exitcode} );
}
}
}
}
if ( ( $a == 1 ) ( $_=~
m/^\s+number\s+of\s+sites\s+\-\\s+([0-9]+)$/ ) ) {
$a=2;# sites
${sites}=$1;
}
if ( ( $a == 0 ) ( $_=~ m/^(.+)$/ ) ) {
$a=1;# volume name
${volume}=$1;
}
if ( $_=~ m/^$/ ) {
$a=0;# new block
}
}
close ( FILE );
# log end
write_log( INFO, OpenAFS replication end );
}
exit ( main );
--
With kind regards,
Marcel
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] Behalf Of Adam Megacz
Sent: 07 February 2008 19:22
To: openafs-info@openafs.org
Cc: [EMAIL PROTECTED]
Subject: [OpenAFS] byte-level incremental backups using full+reverse
delta
In case anybody else finds this useful, I've worked out a system for
doing backups of an AFS volume-set with the full and reverse delta
style of incremental backups.
/afs/megacz.com/srv/bin/dump.sh
Much like rdiff-backup, this keeps a complete copy of the most recent
backup, and stores all previous backups as deltas against the
*following* backup (ie in the reverse direction). This means you
can truncate the backup history whenever you like, rather than only
at full-backup intervals, and there's never any reason to keep more
than a single full backup around.
Deltas are done with xdelta3. Although it is very CPU-intensive, it
seems to be very good at representing minor changes (ie a few bytes)
to very large files, which incremental dumps cannot do, and which
(in my experience) rdiff did not do as well as I would like. A test
restore of each delta is done before deleting the old volume, so you
don't need to trust that the xdelta3 algorithm is correct -- you just
have to trust it that it is deterministic.
Hope you find this useful,
- a
#!/bin/bash -e
# A script for full and backward diff style incremental backups of
# one or more cells' AFS dumpfiles using xdelta3 for diffing. Note
# that xdelta3 will efficiently represent minor changes to very large
# files, which AFS incremental dumps cannot do.
#
# IMPORTANT: you must use xdelta3 version SVN.227 or later -- this
# will become release 3.0u at some point. There is a precompiled x86
# deb at /afs/megacz.com/debian/xdelta3/xdelta3_svn227.deb
#
# Change these variables to suit your needs. Note:
# - Backups are kept in $BACKUPDIR/year/month/day/cell/vol.afsdump.
# - At all times the latest backup is kept in full form, and all
# previous backups are kept as reverse diffs against the backup
# from the day AFTER them. This lets you easily truncate the
# backup history at any time.
# - The symlink $BACKUPDIR/yesterday points to yesterday's backups, if
# there were any; upon completion of today's backups,
Re: [OpenAFS] OpenAFS on windows - profile in AFS, who uses it?
Stephen Joyce wrote: 6. Profiles saved under Windows contain unicode characters. Profile and redirected folders don't currently work well with the AFS client because it doesn't support unicode file names. This isn't a huge issue, but sometimes this comes into play for our foreign students who save web browser items in AFS, or IE does with the history, etc. Sometimes in our environment the profile of a user will not save/load because the filename has strange characters in it. Our help desk ends up fixing each of these users by hand. We've seen the same problem; again, usually with ESL users. It's slightly worse than Rodney said: the unicode files are created on the local disk, then during the sync at logoff, the unicode characters are converted to question marks in the filenames in AFS. The next time the user tries to log in, Windows refuses to copy the files because ? is an illegal character in Windows filenames. The user then can't log in (if you have it set so that profiles must be loaded), or gets a temp profile (if you allow logins when roaming profiles aren't available.) We're still using a fairly old version of the Windows client, but if this is still the behavior, I'd like to see it changed (like using a Windows-legal character rather than ? to replace unicode characters when saving into AFS). Stephen, please remember that the Windows AFS client does not support UNICODE in its implementation of the CIFS protocol. The UNICODE names are never given to the AFS client. The UNICODE to OEM charset translation is performed by the Windows CIFS client. If you have a complaint regarding the behavior of the Windows CIFS client, please file a report with Microsoft. Microsoft must hear the complaints from users, not software authors. By the end of the year this should no longer be an issue because I hope to have the conversion to a native redirector based file system completed by then. Jeffrey Altman Secure Endpoints Inc. smime.p7s Description: S/MIME Cryptographic Signature
RE: [OpenAFS] byte-level incremental backups using full+reverse delta
These contributions are appreciated, especially if someone can use them.
But anytime you send something like this to a list or put it on the web,
it's a very good idea to state the license terms in the script, or at least
in the accompanying message. Unless you release your work into the public
domain or under an open license, some people may hesitate to use it and may
not know whether or not they can re-distribute your work, and if so under
what conditions.
I recommend http://www.gnu.org/licenses/licenses.html :-)
Cheers, Stephen
--
Stephen Joyce
Systems AdministratorP A N I C
Physics Astronomy Department Physics Astronomy
University of North Carolina at Chapel Hill Network Infrastructure
voice: (919) 962-7214and Computing
fax: (919) 962-0480 http://www.panic.unc.edu
Don't judge a book by its movie.
On Fri, 8 Feb 2008, Marcel Koopmans wrote:
Well as we are giving away scripting for replication...
This script works *without* configuration and maybe people can learn from
it.
Just run it on volservers and don't forget to get your AFStoken first.
repl_afs.pl
--
#!/usr/bin/perl -w
#
# OpenAFS volume replication
# by : Marcel D.A. Koopmans of Elysium Open Systems
#
# version : 1.0.5 ( 2007-08-24 20:25 GMT+1 )
use strict;
use Sys::Syslog;
sub write_log {
my ( ${level}, ${message} ) = @_;
openlog( afs_repl, , DAEMON );
syslog( ${level}, ${message} .\n );
closelog;
}
sub main {
my ${err}=0;
my ( $a, $b );
my @temp;
my ( @ip_a, %ip_h );
my ( %hn_h );
my ( $volume, $sites, $host );
my ( ${exitcode} );
# log start
write_log( INFO, OpenAFS replication start );
# get local IP addresses
open ( FILE, /sbin/ifconfig -a | );
while ( FILE ) {
chomp($_);
if ( $_=~ m/^\s+inet\s+addr:([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\s+.*$/ ) {
if ( ! exists ( ${ip_h{$1}} ) ) {
${ip_h{$1}}=1;
${ip_a[$#{ip_a}+1]}=$1;
}
}
}
close ( FILE );
# sort IP addresses;
@[EMAIL PROTECTED];
@ip_a=sort(@temp);
$#temp=-1;
# resolve hostnames
if ( $#{ip_a} -1 ) {
for ( $a=0; $a=$#{ip_a}; $a++ ) {
open ( FILE, /usr/bin/getent hosts . ${ip_a[$a]} . | );
while ( FILE ) {
chomp($_);
@temp=split(/\s+/,$_);
for ( $b=0; $b=$#temp; $b++ ) {
if ( ! exists ( ${hn_h{$temp[$b]}} ) ) {
${hn_h{$temp[$b]}}=1;
}
}
}
close ( FILE );
}
}
# Get openAFS volumes
$a=-1;
open ( FILE, /usr/bin/vos listvldb 2 /dev/null | );
while ( FILE ) {
chomp($_);
$_=~ s/\s+$//;
if ( ( $a == 2 ) ( $_=~
m/^\s+server\s+(.+)\s+partition\s+\/vicep[a-z]+\s+RW\s+Site$/ ) ) {
${host}=$1;
if ( exists ( $hn_h{$host} ) ) {
if ( ${sites} 1 ) {
${exitcode}=system(/usr/bin/vos release . ${volume} .
/dev/null 21 );
${exitcode}=${exitcode} / 256;
if ( ${exitcode} == 0 ) {
write_log( INFO, release of volume . ${volume} .
success );
} else {
write_log( ERR, release of volume . ${volume} . failed,
returned exit code . ${exitcode} );
}
}
}
}
if ( ( $a == 1 ) ( $_=~
m/^\s+number\s+of\s+sites\s+\-\\s+([0-9]+)$/ ) ) {
$a=2;# sites
${sites}=$1;
}
if ( ( $a == 0 ) ( $_=~ m/^(.+)$/ ) ) {
$a=1;# volume name
${volume}=$1;
}
if ( $_=~ m/^$/ ) {
$a=0;# new block
}
}
close ( FILE );
# log end
write_log( INFO, OpenAFS replication end );
}
exit ( main );
--
With kind regards,
Marcel
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] Behalf Of Adam Megacz
Sent: 07 February 2008 19:22
To: openafs-info@openafs.org
Cc: [EMAIL PROTECTED]
Subject: [OpenAFS] byte-level incremental backups using full+reverse
delta
In case anybody else finds this useful, I've worked out a system for
doing backups of an AFS volume-set with the full and reverse delta
style of incremental backups.
/afs/megacz.com/srv/bin/dump.sh
Much like rdiff-backup, this keeps a complete copy of the most recent
backup, and stores all previous backups as deltas against the
*following* backup (ie in the reverse direction). This means you
can truncate the backup history whenever you like, rather than only
at full-backup intervals, and there's never any reason to keep more
than a single full backup around.
Deltas are done with xdelta3. Although it is very CPU-intensive, it
seems to be very good at representing minor changes (ie a few bytes)
to very large files, which incremental dumps cannot do, and which
(in my experience) rdiff did not do as well as I would like. A test
restore of each delta is done before deleting the old volume, so you
don't need to trust that the xdelta3 algorithm is correct -- you just
have to trust it that it is deterministic.
Hope you find this useful,
- a
#!/bin/bash -e
# A
Re: [OpenAFS] OpenAFS on windows - profile in AFS, who uses it?
I second almost everything Rodney said. But I have a few comments (inline). On Sun, 10 Feb 2008, Rodney M. Dyer wrote: 2. A users profile has a folder under it called Local Settings. THIS FOLDER DOES NOT ROAM. This folder only exists during your session on the local machine. When you logout, the data in that folder is considered temporary for your session. Microsoft in further grand wisdom decided to store valuable information in that folder that you really need to carry around with you with the profile, but this data is excluded by default. Notable application data includes: Microsoft Outlook email settings and PST files, etc.. Microsoft IE history, etc.. Microsoft Visual Studio .NET option settings,etc. This is a very good reason to recommend Firefox and Thunderbird. The most annoying thing for my users was that the desktop picture is a Local Setting that doesn't roam. Clever logout and login scripts took care of this though. 6. Profiles saved under Windows contain unicode characters. Profile and redirected folders don't currently work well with the AFS client because it doesn't support unicode file names. This isn't a huge issue, but sometimes this comes into play for our foreign students who save web browser items in AFS, or IE does with the history, etc. Sometimes in our environment the profile of a user will not save/load because the filename has strange characters in it. Our help desk ends up fixing each of these users by hand. We've seen the same problem; again, usually with ESL users. It's slightly worse than Rodney said: the unicode files are created on the local disk, then during the sync at logoff, the unicode characters are converted to question marks in the filenames in AFS. The next time the user tries to log in, Windows refuses to copy the files because ? is an illegal character in Windows filenames. The user then can't log in (if you have it set so that profiles must be loaded), or gets a temp profile (if you allow logins when roaming profiles aren't available.) We're still using a fairly old version of the Windows client, but if this is still the behavior, I'd like to see it changed (like using a Windows-legal character rather than ? to replace unicode characters when saving into AFS). But I digress. Regardless, this mess is another good reason to recommend Firefox. It stores a user's history and bookmarks in single (intelligently-named) files rather than in separate files with names reflecting page titles. Cheers, Stephen -- Stephen Joyce Systems AdministratorP A N I C Physics Astronomy Department Physics Astronomy University of North Carolina at Chapel Hill Network Infrastructure voice: (919) 962-7214and Computing fax: (919) 962-0480 http://www.panic.unc.edu Don't judge a book by its movie. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] OpenAFS on windows - profile in AFS, who uses it?
Stephen Joyce [EMAIL PROTECTED] wrote: On Sun, 10 Feb 2008, Rodney M. Dyer wrote: 2. A users profile has a folder under it called Local Settings. THIS FOLDER DOES NOT ROAM. This folder only exists during your session on the local machine. When you logout, the data in that folder is considered temporary for your session. Microsoft in further grand wisdom decided to store valuable information in that folder that you really need to carry around with you with the profile, but this data is excluded by default. Notable application data includes: Microsoft Outlook email settings and PST files, etc.. Microsoft IE history, etc.. Microsoft Visual Studio .NET option settings,etc. This is a very good reason to recommend Firefox and Thunderbird. The most annoying thing for my users was that the desktop picture is a Local Setting that doesn't roam. Clever logout and login scripts took care of this though. Last time I checked, attempting to use an Outlook PST file from ANY network file system was considered unsafe, which is probably why it defaults to a local folder. CDC ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
