Issues with network file systems and CVS
We've been using CVS with the repository exported via NFS to our UNIX boxen. now we need to connect to another cvs repo at a remote site and the only access the want to give us is via wincvs with the repository on a mapped drive. Now, just from discussion on this list I realise this is bad. My question is what are the potential issues with this approach? I need a better answer than I've heard it's bad to try and cut through the fog of developer inertia and blinding stupidity we've got here, not that It'll help. Could someone explain to me the issues with mounting the repo via a network filesystem? -- [EMAIL PROTECTED] Matthew Navarre It was a hard sell, since he's a database person, and as far as I've seen, once those database worms eat into your brain, it's hard to ever get anything practical done again. To a database person, every nail looks like a thumb. Or something like that. - jwz ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
We've been using CVS with the repository exported via NFS to our UNIX boxen. This seems to cause an undue amount of repository file corruption, and should be avoided. now we need to connect to another cvs repo at a remote site and the only access the want to give us is via wincvs with the repository on a mapped drive. What do you mean with the repository on a mapped drive? If it's at a remote site, use some sort of client-server system, either ssh or pserver depending on your needs. This should be much less intrusive than somehow NFS-mounting the drive from afar. Now, just from discussion on this list I realise this is bad. My question is what are the potential issues with this approach? I need a better answer than I've heard it's bad to try and cut through the fog of developer inertia and blinding stupidity we've got here, not that It'll help. Could someone explain to me the issues with mounting the repo via a network filesystem? In the first place, it exposes the repository to anything anybody might do to it, whereas client-server restricts what somebody can do to the repository while making a mistake. In the second place, it seems to cause data corruption. This is probably a case of locking problems, so that more than one server process will be working on the same directory without realizing it. -- Now building a CVS reference site at http://www.thornleyware.com [EMAIL PROTECTED] ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
RE: Setting up Email Notification
Which OS is the server running?
My server is Windows 2000 professional. I use the cygwin tools for all my
scripting. http://www.cygwin.com/
They have an ssmtp client which I use for mail. I wrap it in the following
mail bash script:
#!/bin/bash
from=[EMAIL PROTECTED]
from=$USERNAME
subject=mail test
html=
test=
send_mail()
{
typeset line
printf To: $to_list\n
printf From: $from\n
printf Subject: $subject\n
if test $html !=
then
printf Content-type: text/html;\n\n
printf html\nbody\n\n
else
printf \n
fi
cat
if test $html !=
then
printf \n/body\n/html\n
fi
return
while read line
do
if test $line = .
then
return
else
printf $line\n
fi
done
}
#set -x
while test ${1:0:1} = -
do
case $1 in
-s )
subject=$2
shift
shift ;;
-f )
from=$2
shift
shift ;;
-html )
html=1
shift ;;
-test )
test=1
shift ;;
esac
done
if test $* =
then
echo usage: $0 [ -s subjectline ] [ -s sender ] [ -html ] [ -test ]
recipient [ recipient . . . ]
exit
fi
to_list=$*
if test $test !=
then
#send_mail | sed 's/\n/\r\n/g' | tee | /usr/sbin/ssmtp $to_list
send_mail | sed 's/\n/\r\n/g'
else
send_mail | sed 's/\n/\r\n/g' | /usr/sbin/ssmtp $to_list
fi
###
This script can be used
echo the include file changed | mail -s Include File Change Notice
[EMAIL PROTECTED]
I usually take the to-list from the contents of CVSROOT/writers.
ssmtp requires a config file /etc/ssmtp/ssmtp.conf to be set up to point to
your smtp server.
Bill
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of
Riechers, Matthew W
Sent: Monday, September 16, 2002 09:05
To: 'the_witchman'
Cc: [EMAIL PROTECTED]
Subject: RE: Setting up Email Notification
From: the_witchman [mailto:[EMAIL PROTECTED]]
Has anyone successfully set up email notification on a CVS server that
doesn't have a mail server?
You could probably send messages via command-line email client (assuming it
had access to a mail server).
-Matt
___
Info-cvs mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/info-cvs
___
Info-cvs mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
On Tue, 17 Sep 2002 11:57:30 -0700, Matthew Navarre [EMAIL PROTECTED] asked: We've been using CVS with the repository exported via NFS to our UNIX boxen. now we need to connect to another cvs repo at a remote site and the only access the want to give us is via wincvs with the repository on a mapped drive. Now, just from discussion on this list I realise this is bad. My question is what are the potential issues with this approach? I need a better answer than I've heard it's bad to try and cut through the fog of developer inertia and blinding stupidity we've got here, not that It'll help. Could someone explain to me the issues with mounting the repo via a network filesystem? The format of the rcs ,v file does not include any kind of a checksum, so any corruptions that occur across a network filesystem are not detected. Corruptions in the past that I have persionally witnessed include: - automagic insertion of carriage-return characters before every newline (eg, writing in text rather than binary mode) - replacing a block with NUL bytes rather than the delta information or text file information. In the case of the corruption of the delta information, this meant that past versions of the file were not available and this was not detected for a long time. In another intance, it was later found that a CLIENT machine didn't have UDP checksums enabled and that there was a lot of network in that part of the network. This caused corrupted NFS packets to be used when writing the ,v file. This particular problem does not happen when cvs is used remotely as cvs takes care to make sure the packet is intact and correct on both sides of the connection. - various udp packets being lost to properly create or delete the #cvs.rfl or #cvs.wfl lock files for the directory caused some stale lock files to be created in some cases and some multiple write operations to occur in the same directory at the same time It is also the case that going to a central server to do the update of your cvs files allows a process local to the cvs server to determine that the network connection to its parent on the remote machine has disappeared so that it can properly remove the cvs locks it has created. For the above reasons, I strongly recommend that you do NOT use NFS to update your repository. I do not have any personal experience with using samba as the remote filesystem, but the remote CVS capability is easy to use and very reliable. I hope the information is of use to you. Enjoy! -- Mark ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
At 02:57 PM 9/17/2002, Matthew Navarre wrote: Could someone explain to me the issues with mounting the repo via a network filesystem? Here's what happened to us: a file was corrupted and we could not recover older versions. (Our backup administrator failed to verify that the backup system was in fact backing up the disk.) It delayed our development for nearly a day while we tried to recover. Do you want something like this to happen when a deadline is looming? Even if your backup system is good? I thought not! Use client-server CVS. The technical reason for the failure has to do with locking in the repository. Locking is a tricky thing to do in a distributed system. Implementations often have subtle bugs. Fred ___ Frederic W. Brehm, Sarnoff Corporation, http://www.sarnoff.com/ ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
On Tuesday 17 September 2002 12:40 pm, [EMAIL PROTECTED] wrote: We've been using CVS with the repository exported via NFS to our UNIX boxen. This seems to cause an undue amount of repository file corruption, and should be avoided. Yah, I know. I've been trying to change it to using ssh, but keep getting but... it works now from TPTB. now we need to connect to another cvs repo at a remote site and the only access the want to give us is via wincvs with the repository on a mapped drive. What do you mean with the repository on a mapped drive? If it's at a remote site, use some sort of client-server system, either ssh or pserver depending on your needs. This should be much less intrusive than somehow NFS-mounting the drive from afar. That's why I'm basically collecting ammo. They want us to use wincvs with a SMB mounted repo. Since I use FreeBSD this presents something of a problem. Could someone explain to me the issues with mounting the repo via a network filesystem? In the first place, it exposes the repository to anything anybody might do to it, whereas client-server restricts what somebody can do to the repository while making a mistake. In the second place, it seems to cause data corruption. This is probably a case of locking problems, so that more than one server process will be working on the same directory without realizing it. Thanks. That's what I need to try to convince them to DTRT. Thanks also to everyone else who replied. More ammo's always good :) -- [EMAIL PROTECTED] Matthew Navarre It was a hard sell, since he's a database person, and as far as I've seen, once those database worms eat into your brain, it's hard to ever get anything practical done again. To a database person, every nail looks like a thumb. Or something like that. - jwz ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
At 04:10 PM 9/17/2002, Matthew Navarre wrote: That's why I'm basically collecting ammo. They want us to use wincvs with a SMB mounted repo. Since I use FreeBSD this presents something of a problem. Bad, bad, very bad idea. I hope you have enough ammo, now. Have a CVS server running on the same _physical_ system that has the disk drives with the repository. Use CVS clients (e.g., WinCvs on Windows, cvs on FreeBSD) to operate on the repository. Fred ___ Frederic W. Brehm, Sarnoff Corporation, http://www.sarnoff.com/ ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
Frederic Brehm writes: The technical reason for the failure has to do with locking in the repository. Locking is a tricky thing to do in a distributed system. Implementations often have subtle bugs. A number of people have now said that and, while it may be true, it doesn't apply to CVS. CVS's locking scheme ensures that only one process is writing to a repository directory at a time, even across network filesystems. The only known potential failure mode, which has never been reported as actually occurring, results in *eveyone* being locked out of a directory, not allowing two servers to write at the same time. No, the problem is just that network filesystems, particularly those that run over unreliable connections like NFS traditionally does, are extremely tricky to get right. They're even trickier when you don't control both ends of the connection, as when the client software is supplied by a different vendor than the server software. All the problems I've seen seem to be either handshaking bugs -- where the client and the server don't communicate reliably -- or simple housekeeping bugs in the server -- where data intended for the disk never actually gets there. Why CVS should be so good at triggering these problems, I don't know. I do know that the commercial CAD/CAM/CAE system my employer produces is also good at triggering them, though. -Larry Jones These pictures will remind us of more than we want to remember. -- Calvin's Mom ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
Matthew Navarre writes: That's why I'm basically collecting ammo. They want us to use wincvs with a SMB mounted repo. Since I use FreeBSD this presents something of a problem. Perhaps you should just recruit someone to hack in and destroy their repository or collect some sensitive information and send it to them -- SMB is not noted for its security. :-) -Larry Jones I told her to expect you to deny everything. -- Calvin ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
At 04:54 PM 9/17/2002, Larry Jones wrote: Frederic Brehm writes: The technical reason for the failure has to do with locking in the repository. Locking is a tricky thing to do in a distributed system. Implementations often have subtle bugs. A number of people have now said that and, while it may be true, it doesn't apply to CVS. CVS's locking scheme ensures that only one process is writing to a repository directory at a time, even across network filesystems. The only known potential failure mode, which has never been reported as actually occurring, results in *eveyone* being locked out of a directory, not allowing two servers to write at the same time. But, isn't a network filesystem a distributed system? I wasn't referring to CVS by itself as a distributed system with the bug. It's the distributed filesystem that CVS uses where the bugs are. No, the problem is just that network filesystems, particularly those that run over unreliable connections like NFS traditionally does, are extremely tricky to get right. They're even trickier when you don't control both ends of the connection, as when the client software is supplied by a different vendor than the server software. All the problems I've seen seem to be either handshaking bugs -- where the client and the server don't communicate reliably -- or simple housekeeping bugs in the server -- where data intended for the disk never actually gets there. Why CVS should be so good at triggering these problems, I don't know. I do know that the commercial CAD/CAM/CAE system my employer produces is also good at triggering them, though. Yup. We agree. It's the filesystem. I suppose that we could get into a semantic argument where we quibble over whether CVS+NFS is an example of a distributed system. But, let's not go there. Fred ___ Frederic W. Brehm, Sarnoff Corporation, http://www.sarnoff.com/ ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
On Tuesday 17 September 2002 01:56 pm, [EMAIL PROTECTED] wrote: Matthew Navarre writes: That's why I'm basically collecting ammo. They want us to use wincvs with a SMB mounted repo. Since I use FreeBSD this presents something of a problem. Perhaps you should just recruit someone to hack in and destroy their repository or collect some sensitive information and send it to them -- SMB is not noted for its security. :-) Well, since it's another office in the same buisness group here I'd pretty much be shooting myself in the foot doing that :) We have a VPN with that office, so it's not unencrypted SMB over the public internet. But i get your point. -Larry Jones I told her to expect you to deny everything. -- Calvin Did you pick that quote manually, or is your sigmonster just unnaturally perceptive? :) -- [EMAIL PROTECTED] Matthew Navarre It was a hard sell, since he's a database person, and as far as I've seen, once those database worms eat into your brain, it's hard to ever get anything practical done again. To a database person, every nail looks like a thumb. Or something like that. - jwz ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
This discussion is interesting to me because of the way we are setting up our CVS and working directories to function. I hope the diagrams draw nice for everyone. 1. I have been told that the following setup will lead to corruption because of esoteric problems in SMB that CVS can bring out: --- -- |Linux CVS| |Windows | |Server | |Workstation | | --- | || | | | C P | | | -- | | | Repo | V S | | | | WinCVS | | | | | S E |--CVS protocol--| Client | | | | | R | | | ^- | | | | V | | | | | | | | E | | | v- | | | | R | | | | Mapped | | | --- | | | Drive | | | | | S S | | | ^- | | | Work | M H | | | | | | | Dir | B A |--SMB protocol | | | | R | | || | | | E | | -- | --- | --- Fine. Won't do that. 2. Is this setup any better or does it still have the SMB issues? --- -- |Linux CVS| |Windows | |Server | |Workstation | | --- | || | | | C P | | | -- | | | Repo | V S | | | | WinCVS | | | | | S E |--CVS protocol--| Client | | | | | R | | | ^- | | | | V | | | | | | | | E | | | v- | | | | R | | | | Mapped | | | --- | | | Drive | | --- | ^- | --|--- ---| | Linux or Win|| | File Server || | --- || | | | S S | || | | Work | M H | || | | Dir | B A |--SMB protocol | | | R | | | | | E | | | --- | --- 3. So the only reliable operation is to have the working directory on a locally mounted file system? --- -- |Linux CVS| |Windows | |Server | |Workstation | | --- | || | | | C P | | | -- | | | Repo | V S | | | | WinCVS | | | | | S E |--CVS protocol--| Client | | | | | R | | | ^- | | | | V | | | | | | | | E | | | v- | | | | R | | | | Work | | | --- | | | Dir| | --- | -- | -- Alan ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
AFIAK the issue is not with where the working copy is stored, as long as you are the only one accessing the working copy. The issue is where the repository is stored/accessed from. Replacing the CVS protocol that's using pserver in your diagram with a local protocol accessing the repository via smb is bad, ie: --- -- |Linux CVS| |Windows | |Server | |Workstation | | --- | || | | | C P | | | -- | | | Repo | V S | | | | WinCVS | | | | | S E | | protocol--| Client | | | | | R | | || ^- | | | | V | | C| | | | | | E | | V| v- | | | | R | | S| | Mapped | | | --- | -:local:--| Drive | | | | | S S | | | ^- | | | Work | M H | | | | | | | Dir | B A |--SMB protocol | | | | R | | || | | | E | | -- | --- | --- in other words your CVSROOT shouldn't be: :local:M:\cvs if M is a mounted share. it needs to use pserver, gserver, ssh, rsh, or some other remote protocol. Adam On Tue, 2002-09-17 at 19:28, Alan Dayley wrote: This discussion is interesting to me because of the way we are setting up our CVS and working directories to function. I hope the diagrams draw nice for everyone. 1. I have been told that the following setup will lead to corruption because of esoteric problems in SMB that CVS can bring out: --- -- |Linux CVS| |Windows | |Server | |Workstation | | --- | || | | | C P | | | -- | | | Repo | V S | | | | WinCVS | | | | | S E |--CVS protocol--| Client | | | | | R | | | ^- | | | | V | | | | | | | | E | | | v- | | | | R | | | | Mapped | | | --- | | | Drive | | | | | S S | | | ^- | | | Work | M H | | | | | | | Dir | B A |--SMB protocol | | | | R | | || | | | E | | -- | --- | --- Fine. Won't do that. 2. Is this setup any better or does it still have the SMB issues? --- -- |Linux CVS| |Windows | |Server | |Workstation | | --- | || | | | C P | | | -- | | | Repo | V S | | | | WinCVS | | | | | S E |--CVS protocol--| Client | | | | | R | | | ^- | | | | V | | | | | | | | E | | | v- | | | | R | | | | Mapped | | | --- | | | Drive | | --- | ^- | --|--- ---| | Linux or Win|| | File Server || | --- || | | | S S | || | | Work | M H | || | | Dir | B A |--SMB protocol | | | R | | | | | E | | | --- | --- 3. So the only reliable operation is to have the working directory on a locally mounted file system? --- -- |Linux CVS| |Windows | |Server | |Workstation | | --- | || | | | C P | | | -- | | | Repo | V S | | | | WinCVS | | | | | S E |--CVS protocol--| Client | | | | | R | | | ^- | | | | V | | | | | | | | E | | | v- | | | | R | | | | Work | | | --- | | | Dir| | --- | -- | -- Alan ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
