RE: Changing the native newline mode
On Fri, Feb 12, 2010 at 6:23 PM, Bob Archer bob.arc...@amsi.com wrote: Use the native windows CLI. No clumsy Cygwin needed. But, to each his own. What, CMD? That's an order of magnitude worse than Cygwin. Because it doesn't have the Unix cl tools? If you install msysgit it adds pretty much all the Unix CLI tools to your windows box you can run in CMD. It also gives you a BASH shell if you prefer that. Also, you might want to take a look at Console2... a nice free utility that makes it easier to manage multiple shells on windows and enhances them with better copy/paste support, etc. Twas just a suggestion. You can ignore it as you wish. I would complain to MS about Studio mangling your line endings. Although my understanding was it supported Unix style line endings. If Microsoft fixed problems that were pointed out to them, my life would be so much easier. Unfortunately, I'd have as much luck yelling my issues into the nearest sewer grate and hoping for help from a ninja turtle... Yes and no. MS does fix issues... hence the service packs. Do they fix the issues you care about? Well, if you can prove a lot more people than you care about it, yes. Of course, they WON'T fix your issue if you don't make them aware of it and just bury your head in the sand with comments like the above. Cheers, BOb
Re: Changing the native newline mode
On Sun, Feb 14, 2010 at 2:34 AM, Glenn Maynard gl...@zewt.org wrote: Based on looking through [1] some more, it looks like cp -a wc1 wc2 and renaming working copies should work fine, since the database is inside the working copy, and will just get copied along with the rest. In SVN 1.7 there will be a single .svn folder at the root of a working copy. Beyond 1.7 there are plans to make this configurable so that you could have it in ~/.subversion and shared across all your working copies. Of course the default will be the same as it will be in 1.7. Hopefully there'll still be a way to slice out a piece of a repository (mv wc1/trunk .; rm -rf wc1), which wouldn't work if it's dependent on a global db at the top. There has been talk of adding a svn detach command to do this. Not sure if it will be done as part of 1.7. AFAIK, the plan is to add it later. I have a few gigs of ~5 meg files in Subversion, and the idea of storing large blocks of data in SQLite is a bit scary; I don't think it's designed for blobs that size. Anything that lumps files together like this is effectively subjected to two layers of fragmentation instead of one (filesystem + db). There has never been any plan or discussion to store the pristine files in SQLite. As you point out, it is not well suited for that and would work poorly. SQLite is being used to store the SVN metadata and properties which are arguably just stored in a custom DB today. When the WC data is centralized the current code that has to read all the metadata, parse it and write it back out would be less efficient than using a database and just being update/insert rows as needed. Plus we got some benefits from being able to use SQL indexes. The storage format for the pristine files will still be files but it is being changed to be based on the SHA-1 hash for the files. I'd imagine the structure will be sharded based on the first two characters of the hash. This will bring several benefits: 1) On case insensitive file systems like Windows and OSX it will allow files to be renamed only by case. Today that fails because of the way the pristine copy is stored. Once it is a SHA-1, it will not matter. 2) Space savings. When you have files in a working copy with the same hash, there will only be a single pristine copy stored. This will likely be a minor benefit in 1.7, but imagine when you can have all your working copies centralized in a single location. If you have multiple copies of trunk checked out, or even multiple branches, it is likely there would be a lot of sharing of pristine copies and would save a significant amount of disk space. 3) Performance. This will be a future benefit. But again, imagine you have a single centralized working copy area. When you do a checkout we can enhance the client/server protocol so that the when the server returns the list of items for the client to fetch it also includes the SHA-1. Now the client can be made smart enough to only fetch the items it does not already have. So imagine you have trunk checked out and you want to checkout a branch. Maybe 90% of the files would already be on your disk and the client could just fetch the other 10% and construct the working copy from what it already has available. -- Thanks Mark Phippard http://markphip.blogspot.com/
Re: Changing the native newline mode
On Fri, Feb 12, 2010 at 10:56:28PM -0500, Glenn Maynard wrote: Telling people don't do what you want to do; do what you don't want instead is not helpful. o rly? I haven't been following this thread closely but it seems that your complaint is the classic, Doctor, it hurts when I do this. Several people in the community have suggested, Then don't do that. I would take this sage advice (i.e. don't share working copies across platforms) or at least one of the proposed workarounds (I saw the idea of setting eol-style to CRLF; did someone suggest running unix2dos before using the working copy on your Windows box?) rather than complaining that the advice is not helpful. I can't find it right now but there was a presentation about the top ten ways *not* to use Subvesion. One of the entries was about using Subversion in a way that it was not designed to be used. This arises most frequently when people want to use Subversion as a backup mechanism, but I think it applies equally to your desire to use a single working copy across platforms no matter the cost. good luck, tyler
Re: Changing the native newline mode
On Feb 13, 2010, at 10:11, Tyler Roscoe wrote: I can't find it right now but there was a presentation about the top ten ways *not* to use Subvesion. One of the entries was about using Subversion in a way that it was not designed to be used. This arises most frequently when people want to use Subversion as a backup mechanism, but I think it applies equally to your desire to use a single working copy across platforms no matter the cost. In my experience, sharing a working copy between OSes can work, if you're aware of the limitations. These include: Checking out files with eol-style:native will give them the native line ending style matching the client that did the svn checkout. I am not certain what happens with svn update later: do files get the eol-style of the client currently doing the update or of the client that originally did the checkout? The latter would mean that the native eol-style was stored somewhere in the .svn directories by the client that did the svn checkout. To discover if this is the case, you could check out a working copy on UNIX and check out a working copy from the same URL on Windows and then diff them. If they differ, you may be able to use that information to patch the contents of the .svn directories on your UNIX box on the SMB share, in addition to running a tool like unix2dos or ux2dos to fix the line endings in the actual files. However, this advice will get me yelled at by the list, because we cannot recommend modifying the contents of the .svn directories manually for any reason. So I'm merely saying you could do this; I'm not recommending you do this. Symlinks don't work on Windows. If you check out a working copy on Windows that would contain symlinks on UNIX, you'll instead get a text file, whose contents is link followed by the path to the original item. If you check out on UNIX, you get a symlink. If you check out on UNIX on an SMB share that's hosted on Windows, I'm not sure what you get. Best advice is not to have any symlinks in your repository if you are going to be checking out on Windows ever. Windows and Mac filesystems are by default case-insensitive; UNIX filesystems are usually case-sensitive. If you need to use your repository from Windows or Mac clients, you will want to not commit files to the repository whose names differ only in case. There is a pre-commit hook in the Subversion source distribution that you can install that prevents such things from happening. Windows has several restrictions on the names files can have, which are listed in this Wikipedia article: http://en.wikipedia.org/wiki/Filename#Reserved_characters_and_words You can commit files with these names on UNIX, but trying to move these to Windows will give you a cryptic error. So you should not commit files with these names to the repository. You could write a pre-commit hook to prevent the commit of such files if you believe this will be a frequent problem.
Re: Changing the native newline mode
On Sat, Feb 13, 2010 at 11:11 AM, Tyler Roscoe ty...@cryptio.net wrote: I haven't been following this thread closely but it seems that your complaint is the classic, Doctor, it hurts when I do this. Several people in the community have suggested, Then don't do that. I would take this sage advice (i.e. don't share working copies across platforms) or at least one of the proposed workarounds (I saw the idea of setting eol-style to CRLF; did someone suggest running unix2dos before using the working copy on your Windows box?) rather than complaining that the advice is not helpful. I've explained in reasonable detail why I want to do what I want, and why none of these workarounds work. Saying just don't do that is ignoring everything that's been said. It hurts when I breathe! Then what should you stop doing? On Sat, Feb 13, 2010 at 4:16 PM, Ryan Schmidt subversion-20...@ryandesign.com wrote: Checking out files with eol-style:native will give them the native line ending style matching the client that did the svn checkout. I am not certain what happens with svn update later: do files get the eol-style of the client currently doing the update or of the client that originally did the checkout? The latter would mean that the native eol-style was stored somewhere in the .svn directories by the client that did the svn checkout. To discover if this is the case, you could check out a working copy on UNIX and check out a working copy from the same URL on Windows and then diff them. If they differ, you may be able to use that information to patch the contents of the .svn directories on your UNIX box on the SMB share, in addition to running a tool like unix2dos or ux2dos to fix the line endings in the actual files. However, this advice will get me yelled at by the list, because we cannot recommend modifying the contents of the .svn directories manually for any reason. So I'm merely saying you could do this; I'm not recommending you do this. Even if that worked, I'm pretty sure it would break badly the next time a new directory was created from an update. I'll stick with manually repairing newlines for now. I suspect that it's not a massive project to actually implement this properly; fundamentally, it probably means adjusting svn_wc__get_eol_style to allow substituting native for one of the other EOL modes, based on a working-copy-local setting (not a stored property). When this setting is changed, any affected eol-style:native files would need to be converted (nothing new here; changing eol-style itself does exactly that). The tricky part is in the supporting details: where to put that setting and how to change it, what happens if you svn switch a directory with this setting, etc. Anyhow, while it might be too niche a feature for inclusion in Subversion, it's definitely not outside the overall design, as Tyler suggests. This fits very naturally into Subversion's concept of newline handling. -- Glenn Maynard
Re: Changing the native newline mode
On Sat, Feb 13, 2010 at 7:10 PM, Ryan Schmidt subversion-20...@ryandesign.com wrote: svn export already has a --native-eol switch. This is easy because after exporting, Subversion doesn't have to deal with that directory anymore. But as a fellow user I think it would fit nicely to have the same switch on svn checkout, which would make that working copy always use the indicated native eol-style. Where to store this fact? In Subversion 1.7, working copy metadata will no longer live in .svn directories but in a central sqlite database, so there's probably a place in there where native eol-style could be stored. If Subversion does not already store the native eol-style of the working copy somewhere, this would probably be a good thing to add, regardless of whether we add a mechanism for requesting an alternate eol-style. A database representing the whole working copy? That's odd--I can't think of how that could generally handle actions like cloning a whole WC (cp -a wc1 wc2), pulling a piece out of a WC creating a new WC as a result (mv wc1/trunk .; rm -rf wc1) and renaming a WC (mv wc1 wc1~), all of which work with the current system (and all of which I use with varying frequency). Putting text-base in Sqlite would be unfortunate. One of the great things that could be done with the current format would be to support COW filesystems, which are under active development and hopefully will be fairly common in a few years. That would eliminate the 2x data overhead, while still supporting client-side diffs. I'm not sure that Sqlite is any good at storing large, changing files, either (database fragmentation). (I don't know what the actual design is looking like; I've looked over http://svn.apache.org/repos/asf/subversion/trunk/notes/wc-ng/design, but of course it's rather hard to grasp the overall design from a thirty page design notepad.) I'm not sure we need a mechanism to switch an existing working copy to a different native eol-style. If we do, svn update --native-eol CRLF would be one possible syntax to consider. That's true, and simplifies the idea significantly. I do think it's reasonable to require this be set at checkout time. Another category of use cases comes to mind: svn diff can operate across WC's, so it should function across WC's checked out with different newline styles. (I don't see myself actually trying to implement this--not in the next few months, anyway--so I didn't poke in to see how hard this would be.) However, rather than adding a third function to svn switch, I'd be more in favor of going the other way: reducing switch to a single functionality I'd avoid any design that involves rearranging core features like svn switch, which I suspect would make it a much harder sell to the core developers, who would rightly ask why are we making major interface changes for a minor feature?. -- Glenn Maynard
Re: Changing the native newline mode
On Feb 13, 2010, at 19:00, Glenn Maynard wrote: A database representing the whole working copy? That's odd--I can't think of how that could generally handle actions like cloning a whole WC (cp -a wc1 wc2), pulling a piece out of a WC creating a new WC as a result (mv wc1/trunk .; rm -rf wc1) and renaming a WC (mv wc1 wc1~), all of which work with the current system (and all of which I use with varying frequency). Putting text-base in Sqlite would be unfortunate. One of the great things that could be done with the current format would be to support COW filesystems, which are under active development and hopefully will be fairly common in a few years. That would eliminate the 2x data overhead, while still supporting client-side diffs. I'm not sure that Sqlite is any good at storing large, changing files, either (database fragmentation). (I don't know what the actual design is looking like; I've looked over http://svn.apache.org/repos/asf/subversion/trunk/notes/wc-ng/design, but of course it's rather hard to grasp the overall design from a thirty page design notepad.) You bring up a lot of the same questions I have about the new working copy design. I assume the developers have considered all of this carefully and are making the best choices they can. I plan to wait until 1.7 is released and then just see what's happened. I understand some (all?) of this is already implemented in trunk so you can of course build from there and see how it works already.
Re: Changing the native newline mode
On Feb 13, 2010, at 23:09, Andrey Repin wrote: Second, don't join replies to multiple authors in one post - it's frustrating. Actually, the mailing list guidelines encourage us to do this: http://subversion.apache.org/docs/community-guide/mailing-lists.html#when-to-post
Re: Changing the native newline mode
On Sun, Feb 14, 2010 at 12:09 AM, Andrey Repin anrdae...@freemail.ru wrote: First, I said already, AnkhSVN. And I've already explained that I want to use my existing Linux CLI to manipulate all of my repositories, so I don't have separate SVN interfaces for each of my working copies. That's the *whole point*. A Windows GUI doesn't help me do that at all, nor do any of the other things you've suggested. If I wanted to use a Windows Subversion client, I wouldn't have been here in the first place. Do you think I'm not aware of their existance? On Sun, Feb 14, 2010 at 12:36 AM, Ryan Schmidt subversion-20...@ryandesign.com wrote: Actually, the mailing list guidelines encourage us to do this: And it's fairly standard practice, at least when replying to close neighbors in a thread. -- Glenn Maynard
Re: Changing the native newline mode
On Sat, Feb 13, 2010 at 9:56 PM, C. Michael Pilato cmpil...@collab.net wrote: You can't think how that would handle those actions because many of them won't be handled at all. 'cp -a wc1 wc2' will result in a non-working-copy named 'wc2'. 'mv wc1/trunk .; rm -rf wc1' will result in a non-working-copy named 'trunk'. And so on. There's been talk of adding 'svn' tool support for these actions, of course, but I don't know if the details have been decided upon. Why don't you chime in over on dev@, since that's rather the place to discuss Subversion's development? That's probably redundant--I'm sure all of this is being thought of, but I'll summarize what comes to mind, just in case. Based on looking through [1] some more, it looks like cp -a wc1 wc2 and renaming working copies should work fine, since the database is inside the working copy, and will just get copied along with the rest. (That's the only place I could imagine a working-copy-global database going anyway, else there would be endless problems with finding the database over NFS and in shared directories, knowing when to purge old databases, etc.) Hopefully there'll still be a way to slice out a piece of a repository (mv wc1/trunk .; rm -rf wc1), which wouldn't work if it's dependent on a global db at the top. [1] http://svn.apache.org/repos/asf/subversion/trunk/notes/wc-ng/design And the other part from the earlier mail: Putting text-base in Sqlite would be unfortunate. One of the great things that could be done with the current format would be to support COW filesystems, which are under active development and hopefully will be fairly common in a few years. That would eliminate the 2x data overhead, while still supporting client-side diffs. I'm not sure that Sqlite is any good at storing large, changing files, either (database fragmentation). I have a few gigs of ~5 meg files in Subversion, and the idea of storing large blocks of data in SQLite is a bit scary; I don't think it's designed for blobs that size. Anything that lumps files together like this is effectively subjected to two layers of fragmentation instead of one (filesystem + db). I'm very happy to see the beginnings of svn obliterate support. I brought that up quite a while back, but people at the time seemed certain that there's never a need to remove old data (which I've had to do many times on our CVS repositories due to disk space limits and prevented us from using Subversion for a long time). -- Glenn Maynard
RE: Changing the native newline mode
Is there any way to change the native newline mode for a particular working copy? I'm checking out code in Linux, over a CIFS mount to a Windows machine where it's being used. This is much easier for me than any of the solutions involving Windows clients, but ends up with Unix line endings, leading to newline headaches. This is pretty well explained in the documentation book. Read about the svn:eol-style property. http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.advanced.props.special.eol-style BOb
Re: Changing the native newline mode
On Feb 12, 2010, at 16:05, Bob Archer wrote: Is there any way to change the native newline mode for a particular working copy? I'm checking out code in Linux, over a CIFS mount to a Windows machine where it's being used. This is much easier for me than any of the solutions involving Windows clients, but ends up with Unix line endings, leading to newline headaches. This is pretty well explained in the documentation book. Read about the svn:eol-style property. http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.advanced.props.special.eol-style Bob, I think Glenn's saying that he has set svn:eol-style to native, and he's checking out on a UNIX OS but he wants the eol-style of the files to be native to Windows, not UNIX. Unfortunately, Glenn, I don't see a way to do that. svn export has an option for this, but svn checkout doesn't. --native-eol ARG : use a different EOL marker than the standard system marker for files with the svn:eol-style property set to 'native'. ARG may be one of 'LF', 'CR', 'CRLF' I used to work at a company where we had the same workflow. We decided to never use svn:eol-style native. Instead, we set svn:eol-style to LF, and configured our Windows editors to know how to deal with such files.
RE: Changing the native newline mode
On Feb 12, 2010, at 16:05, Bob Archer wrote: Is there any way to change the native newline mode for a particular working copy? I'm checking out code in Linux, over a CIFS mount to a Windows machine where it's being used. This is much easier for me than any of the solutions involving Windows clients, but ends up with Unix line endings, leading to newline headaches. This is pretty well explained in the documentation book. Read about the svn:eol-style property. http://svnbook.red-bean.com/nightly/en/svn- book.html#svn.advanced.props.special.eol-style Bob, I think Glenn's saying that he has set svn:eol-style to native, and he's checking out on a UNIX OS but he wants the eol-style of the files to be native to Windows, not UNIX. Ah I see. Then wouldn't he just specify svn:eol-style CRLF? Assuming he only every edits with Windows tools. Although I'm not sure why he wouldn't check out with a windows client. I'm not sure how connecting to a Linux machine to checkout to a folder mounted on a windows machine is easier than using the windows CLI. BOb
Re: Changing the native newline mode
On Fri, Feb 12, 2010 at 5:16 PM, Bob Archer bob.arc...@amsi.com wrote: Ah I see. Then wouldn't he just specify svn:eol-style CRLF? Assuming he only every edits with Windows tools. This isn't Windows-only code, and it's not code that only I'm touching. If someone's checking it out in Linux to compile in Linux normally, then it should have Unix line endings as usual. Although I'm not sure why he wouldn't check out with a windows client. I'm not sure how connecting to a Linux machine to checkout to a folder mounted on a windows machine is easier than using the windows CLI. It's easier because it's a fully configured CLI that I already have running all the time, which does everything the way I want. I don't want to have to do half of my CLI work from a clumsy Cygwin window, especially since I have working copies in both Linux and Windows that I deal with simultaneously. Instead, we set svn:eol-style to LF, and configured our Windows editors to know how to deal with such files. Visual Studio just doesn't understand it. It'll load them and do basic editing fine, but copy/paste leads to mixed newlines. -- Glenn Maynard
Re: Changing the native newline mode
On Feb 12, 2010, at 16:32, Glenn Maynard wrote: Although I'm not sure why he wouldn't check out with a windows client. I'm not sure how connecting to a Linux machine to checkout to a folder mounted on a windows machine is easier than using the windows CLI. It's easier because it's a fully configured CLI that I already have running all the time, which does everything the way I want. I don't want to have to do half of my CLI work from a clumsy Cygwin window, especially since I have working copies in both Linux and Windows that I deal with simultaneously. Actually, doesn't cygwin also consider the native eol-style to be LF? I think if you want CRLF line endings, you have to use a real Windows client, not cygwin. Ryan Schmidt wrote: Instead, we set svn:eol-style to LF, and configured our Windows editors to know how to deal with such files. Visual Studio just doesn't understand it. It'll load them and do basic editing fine, but copy/paste leads to mixed newlines. If you can't configure Visual Studio to understand LF line endings, and you cannot switch to an editor that understands LF line endings, then I agree you have a problem. I'm sorry, I don't have any more solutions. I do remember that years ago when we configured UltraEdit to understand LF line endings, there were four separate settings / checkboxes / radio buttons that had to be set a particular way on several different settings screens. It was a nightmare because one of our committers would inevitably forget to set one of the options and begin messing up the line endings.
Re: Changing the native newline mode
On Fri, Feb 12, 2010 at 6:23 PM, Bob Archer bob.arc...@amsi.com wrote: Use the native windows CLI. No clumsy Cygwin needed. But, to each his own. What, CMD? That's an order of magnitude worse than Cygwin. I would complain to MS about Studio mangling your line endings. Although my understanding was it supported Unix style line endings. If Microsoft fixed problems that were pointed out to them, my life would be so much easier. Unfortunately, I'd have as much luck yelling my issues into the nearest sewer grate and hoping for help from a ninja turtle... -- Glenn Maynard
Re: Changing the native newline mode
Greetings, Glenn Maynard! Although I'm not sure why he wouldn't check out with a windows client. I'm not sure how connecting to a Linux machine to checkout to a folder mounted on a windows machine is easier than using the windows CLI. It's easier because it's a fully configured CLI that I already have running all the time, which does everything the way I want. I don't want to have to do half of my CLI work from a clumsy Cygwin window, especially since I have working copies in both Linux and Windows that I deal with simultaneously. Cygwin checkout will give you LF-only line endings, just like what you are doing now. Instead, we set svn:eol-style to LF, and configured our Windows editors to know how to deal with such files. Visual Studio just doesn't understand it. It'll load them and do basic editing fine, but copy/paste leads to mixed newlines. Use AnkhSVN for example. Or native Windows build of Subversion. -- WBR, Andrey Repin (anrdae...@freemail.ru) 13.02.2010, 6:15 Sorry for my terrible english...
Re: Changing the native newline mode
Greetings, Glenn Maynard! Is there any way to change the native newline mode for a particular working copy? I'm checking out code in Linux, over a CIFS mount to a Windows machine where it's being used. This is much easier for me than any of the solutions involving Windows clients, but ends up with Unix line endings, leading to newline headaches. Don't do that. Check out it where it'll be used. -- WBR, Andrey Repin (anrdae...@freemail.ru) 13.02.2010, 6:14 Sorry for my terrible english...
Re: Changing the native newline mode
On Fri, Feb 12, 2010 at 10:15 PM, Andrey Repin anrdae...@freemail.ru wrote: Don't do that. Check out it where it'll be used. Telling people don't do what you want to do; do what you don't want instead is not helpful. First, CMD is quite powerful, if you know how to cook it. It doesn't really matter how powerful it is. I've had fifteen years of practice to make me very efficient in Linux for commandline tasks. That's the reason I have a Linux box next to my Windows one: so I can use each for what they're good at. I'd recompile svn with a manual hack to change its notion of newlines before enduring CMD. -- Glenn Maynard
