Pandora's Black Box (was: Re: Issues with network file systems and CVS)
Eric Siegerman wrote: I'd rephrase there should be no need to depending on your process, you might be able to get away without it. I wouldn't. But I will concede that most of the industry has accepted standards of work in software that could be vastly improved upon, and my should is based on standards higher than those. Or perhaps I just need to think about it some more to see those problems I'm not considering... However, this little exchange has raised an interesting problem for me. Actually, I've bumped up against it frequently lately, but this is the first time I've looked at it from a project management perspective. Let's call it the Pandora's Black Box problem. A developer starts working on an issue (bug, feature, etc.), then decides (after gaining more knowledge of the inner workings of the project) that it *may* be appropriate to take a different tack on the matter. What does she do now? Backbranch(!? - just branch?)) and checkin the WIP from the first attempt, then start the second in a clean sandbox? Leave the (possibly fragile as the main branch evolves) WIP idle and come back to it later (note the DR hazard)? Does anyone have a method in place for dealing with this? I'd be interested to hear... /|/|ike P.S. Yeah, I guess it's OT...sending anyway ;-) ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
At 08:01 PM 9/17/2002, Adam Bregenzer wrote: AFIAK the issue is not with where the working copy is stored, as long as you are the only one accessing the working copy. The issue is where the repository is stored/accessed from. That's correct. I misunderstood the original question and thought it had to do with how the repository was accessed. Letting CVS touch a repository mounted on a network filesystem is a bad idea. Using client-server CVS where the server accesses the repository through a local interface is the recommended method. The diagram by Alan Dayley shows the recommended method. Where you put your sandbox is a different matter. For performance reasons (I/O throughput during compiles and network traffic) you might want to put it on a local disk. Policy makers don't always understand performance issues, though. Fred ___ Frederic W. Brehm, Sarnoff Corporation, http://www.sarnoff.com/ ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
Matthew Navarre writes: Did you pick that quote manually, or is your sigmonster just unnaturally perceptive? :) Randomly selected Calvin and Hobbes quotes have a tendency to be much more pointedly applicable than one would expect. -Larry Jones Moms and reason are like oil and water. -- Calvin ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
Alan Dayley writes: 1. I have been told that the following setup will lead to corruption because of esoteric problems in SMB that CVS can bring out: [client/server CVS with the working directory on a Samba share from the CVS server machine] That won't cause corruption, but lots of people have reported strange permission problems (probably due to the complexity of mapping Windows file permissions to Unix file permissions). 2. Is this setup any better or does it still have the SMB issues? [same thing but with the working directory mounted from a different machine than the CVS server machine] You're still running the same risk of screwy permission problems if you use a Linux server, but a Windows server should be fine (assuming that SMB actually works correctly in all instances). 3. So the only reliable operation is to have the working directory on a locally mounted file system? That's certainly the best choice, at least in terms of reliability and performance. -Larry Jones I can do that! It's a free country! I've got my rights! -- Calvin ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
Frederic Brehm wrote: For performance reasons (I/O throughput during compiles and network traffic) you might want to put it on a local disk. Policy makers don't always understand performance issues, though. It also helps to point out that there should be no need to backup the sandbox directories. This, however, requires that the developers be trained to perform intermediary checkins of work in progress, which itself requires training in branch-and-merge development for larger changes (or just to permit checking in still broken files), so it's not the simplest thing. Nonethteless, things do seem to work the best that way... /|/|ike ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
On Wed, Sep 18, 2002 at 11:23:48AM -0700, Mike Ayers wrote: It also helps to point out that there should be no need to backup the sandbox directories. Well, I wouldn't go that far, for precisely the reasons you describe: This, however, requires that the developers be trained to perform intermediary checkins of work in progress, which itself requires training in branch-and-merge development for larger changes (or just to permit checking in still broken files), so it's not the simplest thing. I'd rephrase there should be no need to depending on your process, you might be able to get away without it. -- | | /\ |-_|/ Eric Siegerman, Toronto, Ont.[EMAIL PROTECTED] | | / The acronym for the powers that be differs in only one letter from that for the pointy-haired boss. ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Issues with network file systems and CVS
We've been using CVS with the repository exported via NFS to our UNIX boxen. now we need to connect to another cvs repo at a remote site and the only access the want to give us is via wincvs with the repository on a mapped drive. Now, just from discussion on this list I realise this is bad. My question is what are the potential issues with this approach? I need a better answer than I've heard it's bad to try and cut through the fog of developer inertia and blinding stupidity we've got here, not that It'll help. Could someone explain to me the issues with mounting the repo via a network filesystem? -- [EMAIL PROTECTED] Matthew Navarre It was a hard sell, since he's a database person, and as far as I've seen, once those database worms eat into your brain, it's hard to ever get anything practical done again. To a database person, every nail looks like a thumb. Or something like that. - jwz ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
We've been using CVS with the repository exported via NFS to our UNIX boxen. This seems to cause an undue amount of repository file corruption, and should be avoided. now we need to connect to another cvs repo at a remote site and the only access the want to give us is via wincvs with the repository on a mapped drive. What do you mean with the repository on a mapped drive? If it's at a remote site, use some sort of client-server system, either ssh or pserver depending on your needs. This should be much less intrusive than somehow NFS-mounting the drive from afar. Now, just from discussion on this list I realise this is bad. My question is what are the potential issues with this approach? I need a better answer than I've heard it's bad to try and cut through the fog of developer inertia and blinding stupidity we've got here, not that It'll help. Could someone explain to me the issues with mounting the repo via a network filesystem? In the first place, it exposes the repository to anything anybody might do to it, whereas client-server restricts what somebody can do to the repository while making a mistake. In the second place, it seems to cause data corruption. This is probably a case of locking problems, so that more than one server process will be working on the same directory without realizing it. -- Now building a CVS reference site at http://www.thornleyware.com [EMAIL PROTECTED] ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
On Tue, 17 Sep 2002 11:57:30 -0700, Matthew Navarre [EMAIL PROTECTED] asked: We've been using CVS with the repository exported via NFS to our UNIX boxen. now we need to connect to another cvs repo at a remote site and the only access the want to give us is via wincvs with the repository on a mapped drive. Now, just from discussion on this list I realise this is bad. My question is what are the potential issues with this approach? I need a better answer than I've heard it's bad to try and cut through the fog of developer inertia and blinding stupidity we've got here, not that It'll help. Could someone explain to me the issues with mounting the repo via a network filesystem? The format of the rcs ,v file does not include any kind of a checksum, so any corruptions that occur across a network filesystem are not detected. Corruptions in the past that I have persionally witnessed include: - automagic insertion of carriage-return characters before every newline (eg, writing in text rather than binary mode) - replacing a block with NUL bytes rather than the delta information or text file information. In the case of the corruption of the delta information, this meant that past versions of the file were not available and this was not detected for a long time. In another intance, it was later found that a CLIENT machine didn't have UDP checksums enabled and that there was a lot of network in that part of the network. This caused corrupted NFS packets to be used when writing the ,v file. This particular problem does not happen when cvs is used remotely as cvs takes care to make sure the packet is intact and correct on both sides of the connection. - various udp packets being lost to properly create or delete the #cvs.rfl or #cvs.wfl lock files for the directory caused some stale lock files to be created in some cases and some multiple write operations to occur in the same directory at the same time It is also the case that going to a central server to do the update of your cvs files allows a process local to the cvs server to determine that the network connection to its parent on the remote machine has disappeared so that it can properly remove the cvs locks it has created. For the above reasons, I strongly recommend that you do NOT use NFS to update your repository. I do not have any personal experience with using samba as the remote filesystem, but the remote CVS capability is easy to use and very reliable. I hope the information is of use to you. Enjoy! -- Mark ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
At 02:57 PM 9/17/2002, Matthew Navarre wrote: Could someone explain to me the issues with mounting the repo via a network filesystem? Here's what happened to us: a file was corrupted and we could not recover older versions. (Our backup administrator failed to verify that the backup system was in fact backing up the disk.) It delayed our development for nearly a day while we tried to recover. Do you want something like this to happen when a deadline is looming? Even if your backup system is good? I thought not! Use client-server CVS. The technical reason for the failure has to do with locking in the repository. Locking is a tricky thing to do in a distributed system. Implementations often have subtle bugs. Fred ___ Frederic W. Brehm, Sarnoff Corporation, http://www.sarnoff.com/ ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
On Tuesday 17 September 2002 12:40 pm, [EMAIL PROTECTED] wrote: We've been using CVS with the repository exported via NFS to our UNIX boxen. This seems to cause an undue amount of repository file corruption, and should be avoided. Yah, I know. I've been trying to change it to using ssh, but keep getting but... it works now from TPTB. now we need to connect to another cvs repo at a remote site and the only access the want to give us is via wincvs with the repository on a mapped drive. What do you mean with the repository on a mapped drive? If it's at a remote site, use some sort of client-server system, either ssh or pserver depending on your needs. This should be much less intrusive than somehow NFS-mounting the drive from afar. That's why I'm basically collecting ammo. They want us to use wincvs with a SMB mounted repo. Since I use FreeBSD this presents something of a problem. Could someone explain to me the issues with mounting the repo via a network filesystem? In the first place, it exposes the repository to anything anybody might do to it, whereas client-server restricts what somebody can do to the repository while making a mistake. In the second place, it seems to cause data corruption. This is probably a case of locking problems, so that more than one server process will be working on the same directory without realizing it. Thanks. That's what I need to try to convince them to DTRT. Thanks also to everyone else who replied. More ammo's always good :) -- [EMAIL PROTECTED] Matthew Navarre It was a hard sell, since he's a database person, and as far as I've seen, once those database worms eat into your brain, it's hard to ever get anything practical done again. To a database person, every nail looks like a thumb. Or something like that. - jwz ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
At 04:10 PM 9/17/2002, Matthew Navarre wrote: That's why I'm basically collecting ammo. They want us to use wincvs with a SMB mounted repo. Since I use FreeBSD this presents something of a problem. Bad, bad, very bad idea. I hope you have enough ammo, now. Have a CVS server running on the same _physical_ system that has the disk drives with the repository. Use CVS clients (e.g., WinCvs on Windows, cvs on FreeBSD) to operate on the repository. Fred ___ Frederic W. Brehm, Sarnoff Corporation, http://www.sarnoff.com/ ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
Frederic Brehm writes: The technical reason for the failure has to do with locking in the repository. Locking is a tricky thing to do in a distributed system. Implementations often have subtle bugs. A number of people have now said that and, while it may be true, it doesn't apply to CVS. CVS's locking scheme ensures that only one process is writing to a repository directory at a time, even across network filesystems. The only known potential failure mode, which has never been reported as actually occurring, results in *eveyone* being locked out of a directory, not allowing two servers to write at the same time. No, the problem is just that network filesystems, particularly those that run over unreliable connections like NFS traditionally does, are extremely tricky to get right. They're even trickier when you don't control both ends of the connection, as when the client software is supplied by a different vendor than the server software. All the problems I've seen seem to be either handshaking bugs -- where the client and the server don't communicate reliably -- or simple housekeeping bugs in the server -- where data intended for the disk never actually gets there. Why CVS should be so good at triggering these problems, I don't know. I do know that the commercial CAD/CAM/CAE system my employer produces is also good at triggering them, though. -Larry Jones These pictures will remind us of more than we want to remember. -- Calvin's Mom ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
Matthew Navarre writes: That's why I'm basically collecting ammo. They want us to use wincvs with a SMB mounted repo. Since I use FreeBSD this presents something of a problem. Perhaps you should just recruit someone to hack in and destroy their repository or collect some sensitive information and send it to them -- SMB is not noted for its security. :-) -Larry Jones I told her to expect you to deny everything. -- Calvin ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
At 04:54 PM 9/17/2002, Larry Jones wrote: Frederic Brehm writes: The technical reason for the failure has to do with locking in the repository. Locking is a tricky thing to do in a distributed system. Implementations often have subtle bugs. A number of people have now said that and, while it may be true, it doesn't apply to CVS. CVS's locking scheme ensures that only one process is writing to a repository directory at a time, even across network filesystems. The only known potential failure mode, which has never been reported as actually occurring, results in *eveyone* being locked out of a directory, not allowing two servers to write at the same time. But, isn't a network filesystem a distributed system? I wasn't referring to CVS by itself as a distributed system with the bug. It's the distributed filesystem that CVS uses where the bugs are. No, the problem is just that network filesystems, particularly those that run over unreliable connections like NFS traditionally does, are extremely tricky to get right. They're even trickier when you don't control both ends of the connection, as when the client software is supplied by a different vendor than the server software. All the problems I've seen seem to be either handshaking bugs -- where the client and the server don't communicate reliably -- or simple housekeeping bugs in the server -- where data intended for the disk never actually gets there. Why CVS should be so good at triggering these problems, I don't know. I do know that the commercial CAD/CAM/CAE system my employer produces is also good at triggering them, though. Yup. We agree. It's the filesystem. I suppose that we could get into a semantic argument where we quibble over whether CVS+NFS is an example of a distributed system. But, let's not go there. Fred ___ Frederic W. Brehm, Sarnoff Corporation, http://www.sarnoff.com/ ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
On Tuesday 17 September 2002 01:56 pm, [EMAIL PROTECTED] wrote: Matthew Navarre writes: That's why I'm basically collecting ammo. They want us to use wincvs with a SMB mounted repo. Since I use FreeBSD this presents something of a problem. Perhaps you should just recruit someone to hack in and destroy their repository or collect some sensitive information and send it to them -- SMB is not noted for its security. :-) Well, since it's another office in the same buisness group here I'd pretty much be shooting myself in the foot doing that :) We have a VPN with that office, so it's not unencrypted SMB over the public internet. But i get your point. -Larry Jones I told her to expect you to deny everything. -- Calvin Did you pick that quote manually, or is your sigmonster just unnaturally perceptive? :) -- [EMAIL PROTECTED] Matthew Navarre It was a hard sell, since he's a database person, and as far as I've seen, once those database worms eat into your brain, it's hard to ever get anything practical done again. To a database person, every nail looks like a thumb. Or something like that. - jwz ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
This discussion is interesting to me because of the way we are setting up our CVS and working directories to function. I hope the diagrams draw nice for everyone. 1. I have been told that the following setup will lead to corruption because of esoteric problems in SMB that CVS can bring out: --- -- |Linux CVS| |Windows | |Server | |Workstation | | --- | || | | | C P | | | -- | | | Repo | V S | | | | WinCVS | | | | | S E |--CVS protocol--| Client | | | | | R | | | ^- | | | | V | | | | | | | | E | | | v- | | | | R | | | | Mapped | | | --- | | | Drive | | | | | S S | | | ^- | | | Work | M H | | | | | | | Dir | B A |--SMB protocol | | | | R | | || | | | E | | -- | --- | --- Fine. Won't do that. 2. Is this setup any better or does it still have the SMB issues? --- -- |Linux CVS| |Windows | |Server | |Workstation | | --- | || | | | C P | | | -- | | | Repo | V S | | | | WinCVS | | | | | S E |--CVS protocol--| Client | | | | | R | | | ^- | | | | V | | | | | | | | E | | | v- | | | | R | | | | Mapped | | | --- | | | Drive | | --- | ^- | --|--- ---| | Linux or Win|| | File Server || | --- || | | | S S | || | | Work | M H | || | | Dir | B A |--SMB protocol | | | R | | | | | E | | | --- | --- 3. So the only reliable operation is to have the working directory on a locally mounted file system? --- -- |Linux CVS| |Windows | |Server | |Workstation | | --- | || | | | C P | | | -- | | | Repo | V S | | | | WinCVS | | | | | S E |--CVS protocol--| Client | | | | | R | | | ^- | | | | V | | | | | | | | E | | | v- | | | | R | | | | Work | | | --- | | | Dir| | --- | -- | -- Alan ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Issues with network file systems and CVS
AFIAK the issue is not with where the working copy is stored, as long as you are the only one accessing the working copy. The issue is where the repository is stored/accessed from. Replacing the CVS protocol that's using pserver in your diagram with a local protocol accessing the repository via smb is bad, ie: --- -- |Linux CVS| |Windows | |Server | |Workstation | | --- | || | | | C P | | | -- | | | Repo | V S | | | | WinCVS | | | | | S E | | protocol--| Client | | | | | R | | || ^- | | | | V | | C| | | | | | E | | V| v- | | | | R | | S| | Mapped | | | --- | -:local:--| Drive | | | | | S S | | | ^- | | | Work | M H | | | | | | | Dir | B A |--SMB protocol | | | | R | | || | | | E | | -- | --- | --- in other words your CVSROOT shouldn't be: :local:M:\cvs if M is a mounted share. it needs to use pserver, gserver, ssh, rsh, or some other remote protocol. Adam On Tue, 2002-09-17 at 19:28, Alan Dayley wrote: This discussion is interesting to me because of the way we are setting up our CVS and working directories to function. I hope the diagrams draw nice for everyone. 1. I have been told that the following setup will lead to corruption because of esoteric problems in SMB that CVS can bring out: --- -- |Linux CVS| |Windows | |Server | |Workstation | | --- | || | | | C P | | | -- | | | Repo | V S | | | | WinCVS | | | | | S E |--CVS protocol--| Client | | | | | R | | | ^- | | | | V | | | | | | | | E | | | v- | | | | R | | | | Mapped | | | --- | | | Drive | | | | | S S | | | ^- | | | Work | M H | | | | | | | Dir | B A |--SMB protocol | | | | R | | || | | | E | | -- | --- | --- Fine. Won't do that. 2. Is this setup any better or does it still have the SMB issues? --- -- |Linux CVS| |Windows | |Server | |Workstation | | --- | || | | | C P | | | -- | | | Repo | V S | | | | WinCVS | | | | | S E |--CVS protocol--| Client | | | | | R | | | ^- | | | | V | | | | | | | | E | | | v- | | | | R | | | | Mapped | | | --- | | | Drive | | --- | ^- | --|--- ---| | Linux or Win|| | File Server || | --- || | | | S S | || | | Work | M H | || | | Dir | B A |--SMB protocol | | | R | | | | | E | | | --- | --- 3. So the only reliable operation is to have the working directory on a locally mounted file system? --- -- |Linux CVS| |Windows | |Server | |Workstation | | --- | || | | | C P | | | -- | | | Repo | V S | | | | WinCVS | | | | | S E |--CVS protocol--| Client | | | | | R | | | ^- | | | | V | | | | | | | | E | | | v- | | | | R | | | | Work | | | --- | | | Dir| | --- | -- | -- Alan ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
