Re: Repository organization for complex project
On 13 October 2010 19:19, David Weintraub qazw...@gmail.com wrote: On Wed, Oct 13, 2010 at 1:42 PM, Les Mikesell lesmikes...@gmail.com wrote: How would you access them if you don't have java/maven/ivy when you want to retrieve a certain version? Before we adopted our Ant projects to use Maven, we simply used the get task. You can always download from a Maven repository using the get and wget commands. Maven repositories, after all, are HTTP accessible. There's no real reason why you can't user wget and curl to do the fetching if you aren't a Java shop. Uploading is trickier though. We simply used a shell script that generated the correct mvn install-file command and didn't have to generate a POM. Maven and Java are fairly easy to install and use open source licenses, so there isn't really a big deal to install them. With the mvn install-file, we didnt' even have to create a pom.xml file. We could build the command on the fly with just a shell script. Our C/C++ guys just use curl to POST the binaries to Nexus over http... we also POST the .pom file and the .md5 and .sha1 files... that is because one of their build toolchain envs cannot have Java on it... Nexus will rebuild the metadata.xml files for you, so all you really need is to post the .pom and the e.g. .so and the .pom.md5, .pom.sha1, .so.md5 and .so.sha1 files and you're done. If you have a java install on the system, use maven or maven-ant-tasks (from ant) or ivy (from ant) to do the post for you. I think I had a set of bash scripts somewhere in http://svn.apache.org/repos/asf/maven/sandbox/trunk/ at some time. I'll see if I can find them But, there's really no reason that you HAVE to use a Maven repository. All you need is a remote storage directory and the concept of version numbers. I'd setup the repository to use Mavenish concept of object with sub-directories for each release and embedd the release into the object between the object's name and suffix. For example, if Project B produced a library libprojb.so, I'd call the thing libprojb-3.2.so, and store it in my projectb/3.2 directory. The advantage of a Maven repository is that you can use a Maven Repository Manager to enforce some good things, such as write-once, authorization and search... but there is nothing stopping you from just copying everything to a big file share in the sky! Imagine if you take the approach of checking into your repository the libprojb.so library for Project A to use. What usually happens is that you have a lib directory under your project with a libprojb.so directory in it. You have no idea what files went into this directory or what version it was from. Two years from now, that libprojb.so directory will still be there and no one will know anything about it. However, imagine if in Project A's make file is a macro PROJB-LIB=projb/3.2, and later on, it uses this information to do a scp from the release repository. You can look at that particular version of Project A and say Ah! It uses version 3.2 of Project B. Since we tagged the source code used for that version of Project B as 3.2, we can still trace down the correct version. And, we can see that the next release of Project A uses version 3.3 of Project B. -- David Weintraub qazw...@gmail.com
Re: Repository organization for complex project
On 13 October 2010 21:42, Les Mikesell lesmikes...@gmail.com wrote: On 10/13/2010 2:52 PM, BRM wrote: From: Les Mikeselllesmikes...@gmail.com We currently commit component binaries back into tags which are then used by other things with external references, and final binaries are managed separately by project, but that seems wrong on several levels. For the projects we do nightly builds on we use an FTP site that the build scripts upload the builds to. This is easy enough to script even on Windows - whether using DOS Batch or Windows Scripting Host. I'm sure Hudson and other build systems can handle that just fine. I'd like to find a generic scheme (and one that can be plugged into hudson if possible) to store build results with a versioning scheme that doesn't force you to keep them forever because most are obsolete after a few new builds but that has a close-coupling to the svn version or at least the tag name of the matching source. But this seems like it should be a common scenario and not something everyone re-invents differently. Well, right now I have a lot of projects that don't have a nightly build - mostly due to how Qt and Visual Studios interact - or rather, the problems in doing so. However, you can do a couple things to track this very well and automatically, partially supported via Subversion's Keyword Substitution. I use $HeadUrl$ in a number of projects to get the URL, which I then parse to get a information about the build; for example, if it's a tag then I grab the release information, or a branch then I build a special branch info to output. For things like the global revision number you'll need to use 'svnversion' to capture the output somehow. The following would probably work for you: 1. Capture the URL somewhere as part of checking it out using $HeadURL$ 2. As part of the build script run and capture the output of svnversion, you might want to do so with svn info too - though this only works if you are NOT doing an export. You can then store these with a copy of the source and resulting binaries to have full traceability. Alternatively, if you are using svn co in the build, just tarball or zip up the project's sandbox at the end of the build script and save it as a whole - this will keep ALL the information, URLs, versions, etc; for you - and is very build system friendly too. This covers the 'backtrack from binary' scenario as long as you keep those extra pieces in the zip with them. But I'm looking more for a filesystem and/or http URL name/path mapping to store the binaries so you can easily find the currently relevant versions in the first place. Use Nexus (A maven repository manager) The backing store is on the file system. It has a searchable interface that can be used over http to, e.g. get the latest version (IIRC even the latest version within a range, e.g [1.0,2.0) will ensure I never get any of the 2.0 artifacts) It's very lightweight and almost trivial to install. Oh and the open source edition is free! I've used the pro edition and it's great but the OSS is fine for most use cases... the only thing I miss from the pro is the staging support -Stephen P.S. I am just a happy Nexus user, I have no affiliation with Sonatype at present. Among other things it should provide a straightforward way to replace our current svn externals to pull specific libraries into place for dependent builds, and also map to our branch/tag name structure for qa and production releases. Maven may be the right answer, but I was thinking in terms of something that could be viewed as a filesystem - maybe like alfresco where additional rules can be imposed. -- Les Mikesell lesmikes...@gmail.com
RE: Subversion on AIX
Linedata Limited Registered Office: 85 Gracechurch St., London, EC3V 0AA Registered in England and Wales No 3475006 VAT Reg No 710 3140 03 -Original Message- From: David Weintraub [mailto:qazw...@gmail.com] Sent: 13 October 2010 18:03 To: Giulio Troccoli Cc: Subversion Subject: Re: Subversion on AIX I was able to build everything until neon. There I get $ ./configure --with-expat=/app/fms/build/lib/libexpat.la --enable-shared=yes --prefix=/app/fms/build checking for a BSD-compatible install... ./install-sh -c checking for gcc... gcc checking for C compiler default output file name... configure: error: in `/app/fms/build/subversion-1.6.13/neon': configure: error: C compiler cannot create executables See `config.log' for more details. cmu...@fmsdwbap01:~/subversion-1.6.13/neon $ Can you give me any help? It looks like it's crapping out when it is trying to determine the default link output. However, apr, apr-util, and expat all worked. I didn't use gcc but cc CC=/usr/vac/bin/cc \ CFLAGS=-qmaxmem=-1 -O2 -qlanglvl=extended \ CPPFLAGS=-I/usr/local/include \ LDFLAGS=-brtl \ ./configure \ --with-expat=/usr/local/lib/libexpat.la \ --enable-shared=yes Can you try that? Adjust paths for expat according to your previous build G
UTF-16 files and inconsistent line endings
Hi I am developing with Visual C++ Express 2008 on Windows. I needed to add some Japanese characters to a source file, whereupon the editor stored the file in UTF-16LE encoding. Subversion (1.6.12) now complains that the file has inconsistent EOL-style. Does Subversion 1.6 support UTF-16LE encoding? Is there a workaround for this, such as setting eol-style to CRLF (i.e. not native)? Best regards David
Re: UTF-16 files and inconsistent line endings
On Thu, Oct 14, 2010 at 09:29:50AM +0100, David Aldrich wrote: Hi I am developing with Visual C++ Express 2008 on Windows. I needed to add some Japanese characters to a source file, whereupon the editor stored the file in UTF-16LE encoding. Subversion (1.6.12) now complains that the file has inconsistent EOL-style. Does Subversion 1.6 support UTF-16LE encoding? Only if you mark the file as binary. Is there a workaround for this, such as setting eol-style to CRLF (i.e. not native)? Mark the file as binary. See http://subversion.tigris.org/issues/show_bug.cgi?id=2194 Stefan
Re: Bug report against SVN 1.6.13
On Thu, Oct 14, 2010 at 01:42:36AM +0200, Paul Maier wrote: Hi there! file b should be read-write here; what do you think: Reproduction script: svn --version svnadmin create xx svn co file:///C:/[...]/xx yy cd yy echo a a svn add a svn propset svn:needs-lock * a svn ci -m svn up svn cp a b ls -lA Observed behaviour: --- File b is read-only. No svn command is available to make file b read-write. svn lock won't do, because file b is not in the repository yet. Even a svn lock a before the copy doesn't help out. Expected behaviour: --- File b is read-write. Files without representation in the repository (files that come from a uncommitted svn cp or svn mv) should be read-write regardless svn:needs-lock setting. I'm fine with this idea if the file is also made read-only after commit. Would you like to try to write a patch against Subversion trunk that implements this feature? (I wouldn't really call it a bug. A bug is when something doesn't work as designed. In this case, it seems like the designers of the lock feature didn't care or overlooked this issue, so the desired behaviour is simply unspecified.) Stefan
Re: UTF-16 files and inconsistent line endings
On Thursday 14 October 2010, David Aldrich wrote: I am developing with Visual C++ Express 2008 on Windows. I needed to add some Japanese characters to a source file, whereupon the editor stored the file in UTF-16LE encoding. Subversion doesn't support UTF-16 as text file, it treats it as unknown (raw binary) file... Subversion (1.6.12) now complains that the file has inconsistent EOL-style. ...and that doesn't mix well with this text-specific setting. You could try using UTF-8 instead, though I'm not sure how to explain that to MSVC, maybe just storing with a BOM is enough. MSVC allows saving as UTF-8 when choosing save as in the save button's drop-down list, notepad.exe does that if you save as UTF-8, IIRC. Is there a workaround for this, such as setting eol-style to CRLF (i.e. not native)? Simply deleting the svn:eol-style property would also silence the warning. Uli -- ML: http://subversion.apache.org/docs/community-guide/mailing-lists.html FAQ: http://subversion.apache.org/faq.html Docs: http://svnbook.red-bean.com/ Sator Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 ** Sator Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 ** Visit our website at http://www.satorlaser.de/ ** Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, weitergeleitet, veröffentlicht oder anderweitig benutzt werden. E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte Änderungen enthalten. Sator Laser GmbH ist für diese Folgen nicht verantwortlich. **
RE: UTF-16 files and inconsistent line endings
Hi Ulrich and Stefan Thanks for your replies. Using UTF-8 has solved the problem, as you suggested. Best regards David -Original Message- From: Ulrich Eckhardt [mailto:eckha...@satorlaser.com] Sent: 14 October 2010 12:23 To: users@subversion.apache.org Subject: Re: UTF-16 files and inconsistent line endings On Thursday 14 October 2010, David Aldrich wrote: I am developing with Visual C++ Express 2008 on Windows. I needed to add some Japanese characters to a source file, whereupon the editor stored the file in UTF-16LE encoding. Subversion doesn't support UTF-16 as text file, it treats it as unknown (raw binary) file... Subversion (1.6.12) now complains that the file has inconsistent EOL-style. ...and that doesn't mix well with this text-specific setting. You could try using UTF-8 instead, though I'm not sure how to explain that to MSVC, maybe just storing with a BOM is enough. MSVC allows saving as UTF-8 when choosing save as in the save button's drop-down list, notepad.exe does that if you save as UTF-8, IIRC. Is there a workaround for this, such as setting eol-style to CRLF (i.e. not native)? Simply deleting the svn:eol-style property would also silence the warning. Uli -- ML: http://subversion.apache.org/docs/community-guide/mailing-lists.html FAQ: http://subversion.apache.org/faq.html Docs: http://svnbook.red-bean.com/ Sator Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 * * Sator Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 * * Visit our website at http://www.satorlaser.de/ * * Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, weitergeleitet, veröffentlicht oder anderweitig benutzt werden. E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte Änderungen enthalten. Sator Laser GmbH ist für diese Folgen nicht verantwortlich. * * Click https://www.mailcontrol.com/sr/j7TZ4!efwiHTndxI!oX7Ug3O71j94L3XM4taKVk7xLgF5u 8fC7h07Llmcw5pzXf0QETRMEibH6EpdY6WjzZSeg== to report this email as spam.
Re: Repository organization for complex project
On Thu, Oct 14, 2010 at 3:39 AM, Stephen Connolly stephen.alan.conno...@gmail.com wrote: Our C/C++ guys just use curl to POST the binaries to Nexus over http... we also POST the .pom file and the .md5 and .sha1 files... that is because one of their build toolchain envs cannot have Java on it... Nexus will rebuild the metadata.xml files for you, so all you really need is to post the .pom and the e.g. .so and the .pom.md5, .pom.sha1, .so.md5 and .so.sha1 files and you're done. I would think that you'd need whatever protocol Maven uses to actually put files on the repository. I guess the Maven protocol is simpler than I thought and simply uses webdav authentication and the mvn file::deploy simply calculates the URL for you. That's good to know. That means you can replace all of Maven with some fairly simple shell scripts using curl. Then in a non-Java project, all the Maven repositories do is provide a nice interface for searching and administration of a HTTP based repository. -- David Weintraub qazw...@gmail.com
RE: Subversion on AIX
I may have already stated this, but the way the build works really sucks. Running configure on AIX powerpc would take 15 to 30 minutes. Each time, determining whether my version of sed truncates characters, the name of my default executable created by the linker, etc. And, when things didn't work, it took hours to trace where the error was occurring and why. The configure shell script is a very long and confusing mess with almost no documentation. The configure script is autogenerated from a couple other files. I believe it is using the standard GNU autoconf stack to generate the config and make files. The original files are M4 scripts (even uglier to look at). Unfortunately, to make the build easier on AIX requires someone with AIX experience working with the GNU autoconf/automake teams. I get the feeling that's hard to come by, which would explain why it's such a pain. On Linux (and probably BSD), the same process is much quicker and is not filled with such nasty problems. In my brief encounters with Solaris, I know there are some big enough differences in the platforms to cause plenty of heartburn. I've done very little with AIX, but it just feels different from both Solarix and Linux or BSD. I shudder to think of the experience using Irix would be like building this... Confidentiality Note: The information contained in this message, and any attachments, may contain proprietary and/or privileged material. It is intended solely for the person or entity to which it is addressed. Any review, retransmission, dissemination, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. Confidentiality Note: The information contained in this message, and any attachments, may contain proprietary and/or privileged material. It is intended solely for the person or entity to which it is addressed. Any review, retransmission, dissemination, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
sparse checkout tool
I remember reading about a tool that let you define what parts of a repository were needed to create a work space. It basically did a bunch of sparse checkouts based on a config file. Could someone remind me what was its name? Please reply directly as I'm no longer on the list. Thanks JLM
Re: sparse checkout tool
On Thu, Oct 14, 2010 at 1:38 PM, Jeremy Mordkoff j...@zeevee.com wrote: I remember reading about a tool that let you define what parts of a repository were needed to create a work space. It basically did a bunch of sparse checkouts based on a config file. Could someone remind me what was its name? Please reply directly as I'm no longer on the list. You must mean this: http://svn.apache.org/viewvc/subversion/trunk/tools/client-side/svn-viewspec.py?view=markup -- Thanks Mark Phippard http://markphip.blogspot.com/
Re: svn Farm
On Thu, Oct 14, 2010 at 10:56:35AM -0700, David Brodbeck wrote: On Sat, Oct 9, 2010 at 6:39 AM, Nico Kadel-Garcia nka...@gmail.com wrote: Second, many working environments in the UNIX world rely on NFS based home directoies, to share working environments and configurations across a variety of machines. In such environments, *any* host that can be leveraged to local root access can su or suco to become the target user, and access their entire home directory. Think I'm kidding? Walk into any university environment: plug in a live Linux CD. Run an nmap scan for hosts running NFS. Run showmount to detect what NFS shares are published to everyone. Go ahead and mount the shares. Look in them for home directoriies. Look in them, using your local root privileges, for Subversion passphrases. (Look for CVS passphrases and un-passphrase-protected SSH keys while you're at it.) This is why running public-facing NFS servers using auth_sys and no_root_squash is a BAD idea. If this is happening at your site, you have much more serious things to worry about than subversion passwords being stolen. For example, in your scenario it would be trivial to create an suid-root shell binary, which a local user could then run and gain root privileges. Exactly. Bad NFS configuration isn't Subversion's fault. Neither are NFS implementations that have insecure default settings, like not mapping 'root' to 'nobody' by default. There are problems with plaintext passwords, no doubt. But the above scenario description is hyped up and misses the point. If you cannot trust root on a UNIX box, don't save anything of value on that box. As of 1.6, Subversion asks the user before saving passwords in plaintext. 1.6 also added support for using GNOME Keyring and KDE Wallet as password stores. It looks like there will be support for using PGP to encrypt passwords soon. Maybe even in 1.7. Some code for this has already entered the repository: http://svn.apache.org/viewvc?view=revisionrevision=1005036 Stefan
Re: Repository organization for complex project
On 10/14/2010 8:24 AM, David Weintraub wrote: On Thu, Oct 14, 2010 at 3:39 AM, Stephen Connolly stephen.alan.conno...@gmail.com wrote: Our C/C++ guys just use curl to POST the binaries to Nexus over http... we also POST the .pom file and the .md5 and .sha1 files... that is because one of their build toolchain envs cannot have Java on it... Nexus will rebuild the metadata.xml files for you, so all you really need is to post the .pom and the e.g. .so and the .pom.md5, .pom.sha1, .so.md5 and .so.sha1 files and you're done. I would think that you'd need whatever protocol Maven uses to actually put files on the repository. I guess the Maven protocol is simpler than I thought and simply uses webdav authentication and the mvn file::deploy simply calculates the URL for you. That's good to know. That means you can replace all of Maven with some fairly simple shell scripts using curl. Then in a non-Java project, all the Maven repositories do is provide a nice interface for searching and administration of a HTTP based repository. Is there a maven-for-dummies reference somewhere that would make a good starting point for how the repository is supposed to work? I tend to get lost easily with java-like things that have a bazillion separate config files all over the place. Also, is mirroring/redundancy built into the design? -- Les Mikesell lesmikes...@gmail.com
Managing modifications to an open source product
I develop for a site that uses Mediawiki (MW). We make some modifications to it before deployment. Generally, (using subversion) we check out a tagged version into a workspace, recursively delete the .svn directories, modify a small number of files, add some of our own extensions, and then commit the result into our own repository. We then work with the source from there. This approach means we have to track MW bug-fixes and add them to our modified version. I was wondering if there is a better way to accomplish the same objective. For example, we can use the svn:externals property to point to the MW repository version of the extensions we use, so each time they are updated, all we need to do is svn up on the externals directory. The main source is a different story. Since we modify some of the files (and have no commit privileges to the MW repository), the files we modify are not within our purview to change (and understandably the MW people wouldn't allow it even if we had commit privileges). Is there any way to use the svn:externals property to solve the main source issue? For example, could we point the revision we keep in our main repository to the correct revision in the MW repository and then tag the appropriate directories that contain the files we modify with svn:external. These latter svn:external properties would name the individual files we modify and point to the modified version that we could keep in our repository. My concern is we are overloading the files in the MW repository with files in our repository and I am not sure subversion allows that. Regards, Dan Nessett
RE: Managing modifications to an open source product
I develop for a site that uses Mediawiki (MW). We make some modifications to it before deployment. Generally, (using subversion) we check out a tagged version into a workspace, recursively delete the .svn directories, modify a small number of files, add some of our own extensions, and then commit the result into our own repository. We then work with the source from there. This approach means we have to track MW bug-fixes and add them to our modified version. I was wondering if there is a better way to accomplish the same objective. For example, we can use the svn:externals property to point to the MW repository version of the extensions we use, so each time they are updated, all we need to do is svn up on the externals directory. The main source is a different story. Since we modify some of the files (and have no commit privileges to the MW repository), the files we modify are not within our purview to change (and understandably the MW people wouldn't allow it even if we had commit privileges). Is there any way to use the svn:externals property to solve the main source issue? For example, could we point the revision we keep in our main repository to the correct revision in the MW repository and then tag the appropriate directories that contain the files we modify with svn:external. These latter svn:external properties would name the individual files we modify and point to the modified version that we could keep in our repository. My concern is we are overloading the files in the MW repository with files in our repository and I am not sure subversion allows that. There is a whole section in the svn book about this... http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.advanced.vendorbr BOb
Re: Managing modifications to an open source product
On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote: I develop for a site that uses Mediawiki (MW). We make some modifications to it before deployment. Generally, (using subversion) we check out a tagged version into a workspace, recursively delete the .svn directories, modify a small number of files, add some of our own extensions, and then commit the result into our own repository. We then work with the source from there. This approach means we have to track MW bug-fixes and add them to our modified version. I was wondering if there is a better way to accomplish the same objective. For example, we can use the svn:externals property to point to the MW repository version of the extensions we use, so each time they are updated, all we need to do is svn up on the externals directory. The main source is a different story. Since we modify some of the files (and have no commit privileges to the MW repository), the files we modify are not within our purview to change (and understandably the MW people wouldn't allow it even if we had commit privileges). Is there any way to use the svn:externals property to solve the main source issue? For example, could we point the revision we keep in our main repository to the correct revision in the MW repository and then tag the appropriate directories that contain the files we modify with svn:external. These latter svn:external properties would name the individual files we modify and point to the modified version that we could keep in our repository. My concern is we are overloading the files in the MW repository with files in our repository and I am not sure subversion allows that. There is a whole section in the svn book about this... http://svnbook.red-bean.com/nightly/en/svn- book.html#svn.advanced.vendorbr BOb I have read this section. It is about vendor drops, but it doesn't answer the question I asked. Basically, we are doing a vendor drop now. Dan -- -- Dan Nessett
Re: Managing modifications to an open source product
On 14/10/2010 21:45, Dan Nessett wrote: On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote: I develop for a site that uses Mediawiki (MW). We make some modifications to it before deployment. Generally, (using subversion) we check out a tagged version into a workspace, recursively delete the .svn directories, modify a small number of files, add some of our own extensions, and then commit the result into our own repository. We then work with the source from there. This approach means we have to track MW bug-fixes and add them to our modified version. I was wondering if there is a better way to accomplish the same objective. For example, we can use the svn:externals property to point to the MW repository version of the extensions we use, so each time they are updated, all we need to do is svn up on the externals directory. The main source is a different story. Since we modify some of the files (and have no commit privileges to the MW repository), the files we modify are not within our purview to change (and understandably the MW people wouldn't allow it even if we had commit privileges). Is there any way to use the svn:externals property to solve the main source issue? For example, could we point the revision we keep in our main repository to the correct revision in the MW repository and then tag the appropriate directories that contain the files we modify with svn:external. These latter svn:external properties would name the individual files we modify and point to the modified version that we could keep in our repository. My concern is we are overloading the files in the MW repository with files in our repository and I am not sure subversion allows that. There is a whole section in the svn book about this... http://svnbook.red-bean.com/nightly/en/svn- book.html#svn.advanced.vendorbr BOb I have read this section. It is about vendor drops, but it doesn't answer the question I asked. Basically, we are doing a vendor drop now. Dan It actually does, but it's a bit hard to understand. Reread it carefully and you'll see that you actually track the public changes in a branch that you later merge back into your trunk.
Re: Repository organization for complex project
What? You want a GOOD Maven manual? Real programmers don't use manuals. We hack away for years in frustration and futility until we die. Maven is one of the WORST Unix documented projects I've seen. I know of only two books: Maven: The Definitive Guide http://www.sonatype.com/books/maven-book/ and Better Builds with Maven http://www.maestrodev.com/support. Better Builds with Maven was full of errors both grammatical and technical. Maven: The Definitive Guide is a bit better. (Do not use Maven, A Developer's Notebook because this is for an obsolete version of Maven. It's like learning Unix administration from a MS-DOS 3.1 manual.) The documentation on the Maven website itself is awful. It looks like someone said We have a Wiki http://docs.codehaus.org/display/MAVENUSER/Home! Documentation complete! Fortunately, you don't really have to know too much about Maven if you're not a Java developer. The intricacies of the pom.xml file don't concern you. Possibly, the only Maven command you really have to know is mvn deploy:deploy-file which doesn't require a pom.xml file http://maven.apache.org/plugins/maven-deploy-plugin/deploy-file-mojo.html. And, if that command concerns you too much, you can simply revert to curl. You do have to understand the Maven repository layout which is a fairly straight forward hierarchal affair and both Nexus and Artifactory cover that pretty well. On Thu, Oct 14, 2010 at 2:42 PM, Les Mikesell lesmikes...@gmail.com wrote: On 10/14/2010 8:24 AM, David Weintraub wrote: On Thu, Oct 14, 2010 at 3:39 AM, Stephen Connolly stephen.alan.conno...@gmail.com wrote: Our C/C++ guys just use curl to POST the binaries to Nexus over http... we also POST the .pom file and the .md5 and .sha1 files... that is because one of their build toolchain envs cannot have Java on it... Nexus will rebuild the metadata.xml files for you, so all you really need is to post the .pom and the e.g. .so and the .pom.md5, .pom.sha1, .so.md5 and .so.sha1 files and you're done. I would think that you'd need whatever protocol Maven uses to actually put files on the repository. I guess the Maven protocol is simpler than I thought and simply uses webdav authentication and the mvn file::deploy simply calculates the URL for you. That's good to know. That means you can replace all of Maven with some fairly simple shell scripts using curl. Then in a non-Java project, all the Maven repositories do is provide a nice interface for searching and administration of a HTTP based repository. Is there a maven-for-dummies reference somewhere that would make a good starting point for how the repository is supposed to work? I tend to get lost easily with java-like things that have a bazillion separate config files all over the place. Also, is mirroring/redundancy built into the design? -- Les Mikesell lesmikes...@gmail.com -- David Weintraub qazw...@gmail.com
Re: Managing modifications to an open source product
On Thu, Oct 14, 2010 at 09:55:08PM +0200, Olivier Sannier wrote: On 14/10/2010 21:45, Dan Nessett wrote: On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote: Generally, (using subversion) we check out a tagged version into a workspace, recursively delete the .svn directories, modify a small number of files, add some of our own extensions, and then commit the result into our own repository. We then work with the source from there. There is a whole section in the svn book about this... http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.advanced.vendorbr I have read this section. It is about vendor drops, but it doesn't answer the question I asked. Basically, we are doing a vendor drop now. It doesn't sound like you're using the approach described in the book. With vendor branching, you work with 2 distinct branches, one containing pristine upstream vendor code, and one containing the modified version. Rather than adding the upstream source code mixed up with your own modifications directly. You might also want to take a look at Piston: http://piston.rubyforge.org/ Stefan
Re: Managing modifications to an open source product
On Thu, 14 Oct 2010 22:33:04 +0200, Stefan Sperling wrote: On Thu, Oct 14, 2010 at 09:55:08PM +0200, Olivier Sannier wrote: On 14/10/2010 21:45, Dan Nessett wrote: On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote: Generally, (using subversion) we check out a tagged version into a workspace, recursively delete the .svn directories, modify a small number of files, add some of our own extensions, and then commit the result into our own repository. We then work with the source from there. There is a whole section in the svn book about this... http://svnbook.red-bean.com/nightly/en/svn- book.html#svn.advanced.vendorbr I have read this section. It is about vendor drops, but it doesn't answer the question I asked. Basically, we are doing a vendor drop now. It doesn't sound like you're using the approach described in the book. With vendor branching, you work with 2 distinct branches, one containing pristine upstream vendor code, and one containing the modified version. Rather than adding the upstream source code mixed up with your own modifications directly. You might also want to take a look at Piston: http://piston.rubyforge.org/ Stefan Yes, you're right. We don't store an exact copy of each drop in our repository. Instead, we copy the drop into a workspace (effectively importing by checking out and then recursively deleting the .svn directories), merge there and commit to the new version. We couldn't understand why we need to keep an exact copy of the vendor drop in our repository. Piston looks interesting, but it also appears to require ruby. This isn't a show stopper, but it would be nice if we could do what we want to do with subversion only. If that isn't possible, or it is too complex, then Piston might be the right way to go. Dan -- -- Dan Nessett
Re: Repository organization for complex project
On 10/14/2010 3:11 PM, David Weintraub wrote: What? You want a GOOD Maven manual? Real programmers don't use manuals. Yeah, I know - they don't write them either (except for subversion, of course). As you might guess, I'm more of a system administrator than a programmer... Fortunately, you don't really have to know too much about Maven if you're not a Java developer. The intricacies of the pom.xml file don't concern you. Possibly, the only Maven command you really have to know is mvn deploy:deploy-file which doesn't require a pom.xml file http://maven.apache.org/plugins/maven-deploy-plugin/deploy-file-mojo.html. And, if that command concerns you too much, you can simply revert to curl. You do have to understand the Maven repository layout which is a fairly straight forward hierarchal affair and both Nexus and Artifactory cover that pretty well. Thanks - the piece I need is how to map our project tags and branch/revision_numbers into the maven repo URLs, hopefully in a way that wouldn't break it for actual maven use since we do have some java developers here. -- Les Mikesell lesmikes...@gmail.com
Re: Managing modifications to an open source product
On Thu, 14 Oct 2010 21:17:59 +, Dan Nessett wrote: On Thu, 14 Oct 2010 21:55:08 +0200, Olivier Sannier wrote: On 14/10/2010 21:45, Dan Nessett wrote: On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote: I develop for a site that uses Mediawiki (MW). We make some modifications to it before deployment. Generally, (using subversion) we check out a tagged version into a workspace, recursively delete the .svn directories, modify a small number of files, add some of our own extensions, and then commit the result into our own repository. We then work with the source from there. This approach means we have to track MW bug-fixes and add them to our modified version. I was wondering if there is a better way to accomplish the same objective. For example, we can use the svn:externals property to point to the MW repository version of the extensions we use, so each time they are updated, all we need to do is svn up on the externals directory. The main source is a different story. Since we modify some of the files (and have no commit privileges to the MW repository), the files we modify are not within our purview to change (and understandably the MW people wouldn't allow it even if we had commit privileges). Is there any way to use the svn:externals property to solve the main source issue? For example, could we point the revision we keep in our main repository to the correct revision in the MW repository and then tag the appropriate directories that contain the files we modify with svn:external. These latter svn:external properties would name the individual files we modify and point to the modified version that we could keep in our repository. My concern is we are overloading the files in the MW repository with files in our repository and I am not sure subversion allows that. There is a whole section in the svn book about this... http://svnbook.red-bean.com/nightly/en/svn- book.html#svn.advanced.vendorbr BOb I have read this section. It is about vendor drops, but it doesn't answer the question I asked. Basically, we are doing a vendor drop now. Dan It actually does, but it's a bit hard to understand. Reread it carefully and you'll see that you actually track the public changes in a branch that you later merge back into your trunk. I just reread the Vendor drop section in the book. It doesn't actually address the question I raised (which is about overloading files obtained with the svn:external definition). We don't do exactly what is described in the book, but logically it is the same. Instead of importing the vendor drop into the repository, we import it into a workspace (manually by checking it out and then recursively deleting the .svn directories. Perhaps we should use the svn import command, but that is a side issue). We then do the merge in the workspace and commit to a new version in the repository. My question has to do with bypassing the fetch of the complete new MW version into our repository. (NB: MW revisions run about 150MB). Let me describe this in more detail. + When we wish to upgrade to a new MW revision, check out the directory in *our* repository that contains our modified files (and only those files). + Merge each of these files with those in the new MW revision. (NB: there are only a very few of these). + After resolving any conflicts, commit these changes into a new revision in *our* repository, probably tagging it so it is easy to reference. + svn mkdir to create an empty workspace. + With svn propedit add an svn:externals property to this directory so that the whole source effectively is an external fetch of the appropriate MW revision. + With svn propedit add lines to the svn:externals definition in the top- level directory that describe the files we just merged, instructing subversion to get these from the new branch we just created in *our* repository. (Note: this is where the overload occurs, since files with the same name and at the same location in the MW repository revision exist. We don't want fetch these - we want a checkout to get the merged files from *our* repository instead.) + Commit this to *our* repository under a new tag. Will this work? If so, are there some reasons why it isn't a good idea. Bad etiquette to answer my own question, but I just reread the Externals Definition section of the book and found this: An externals definition can point only to directories, not to files. So, what I have described won't work even if overloading with svn:externals is supported. -- -- Dan Nessett
Re: Managing modifications to an open source product
Dan, This isn't that much help but are you aware that svn export is exactly the same as svn checkout run a script to delete all the .svn folders ...? The problem with stripping .svn from vendor checkout is then you cannot update it; you're forced to export the whole tree every time. The method of having a working copy somewhere pointing to the vendor files, that you can svn update whenever you like, and then diff the changes into your tree is the easiest way to accomplish what you want, I think. What I would recommend is that you have two branches for this (plus trunk of your code): [your stuff] = trunk-working-copy [their stuff] = from-vendor-svn [intermediate] = svn copy [your stuff -r HEAD] Then at any time you can: svn up [their stuff] svn copy [your stuff] = [intermediate] (so you can do your diff/merge to an intermediate branch) test on [intermediate] if it passes merge it to trunk, delete [intermediate] Bear in mind, not sure what OS you're on, but e.g. on Windows using Araxis merge, merging two or three trees is far (several orders of magnitude) simpler than using svn diff. HTH, Geoff
Re: Managing modifications to an open source product
On Thu, Oct 14, 2010 at 09:17:59PM +, Dan Nessett wrote: My question has to do with bypassing the fetch of the complete new MW version into our repository. (NB: MW revisions run about 150MB). Let me describe this in more detail. + When we wish to upgrade to a new MW revision, check out the directory in *our* repository that contains our modified files (and only those files). + Merge each of these files with those in the new MW revision. (NB: there are only a very few of these). + After resolving any conflicts, commit these changes into a new revision in *our* repository, probably tagging it so it is easy to reference. + svn mkdir to create an empty workspace. + With svn propedit add an svn:externals property to this directory so that the whole source effectively is an external fetch of the appropriate MW revision. + With svn propedit add lines to the svn:externals definition in the top- level directory that describe the files we just merged, instructing subversion to get these from the new branch we just created in *our* repository. (Note: this is where the overload occurs, since files with the same name and at the same location in the MW repository revision exist. We don't want fetch these - we want a checkout to get the merged files from *our* repository instead.) + Commit this to *our* repository under a new tag. Will this work? Unfortunately, this won't work. You cannot use externals like this. The existing files are in the way. But I may be wrong, or maybe I misunderstand your idea. In general, with a complex tool like Subversion the best way to see if something will work is to just try it out and see. If so, are there some reasons why it isn't a good idea. There's nothing wrong with your idea itself. But Subversion doesn't support an overlay concept like you want to realise with externals. Also, I'd recommend not using file externals for anything, because their implementation is a hack and there are several known issues: http://subversion.tigris.org/issues/show_bug.cgi?id=3351 http://subversion.tigris.org/issues/show_bug.cgi?id=3589 http://subversion.tigris.org/issues/show_bug.cgi?id=3665 http://subversion.tigris.org/issues/show_bug.cgi?id=3563 Directory externals are fine though. The problem you're trying to solve (vendor branching) isn't easy, and many have tried before and devised solutions. So I'd suggest that before trying to devise schemes of your own, shop around for existing solutions. One of them will probably fit the bill. The existing alternatives I know about are: - Subversion's vendor branching with svn_load_dirs. This works fine in general, but handling of renamed files is a bit clunky. It's what the svnbook describes. - Play with Subversion's foreign repository merges. This is a little-known feature that could help you track the MediaWiki changes, as long as MediaWiki keeps using Subversion. See http://blogs.open.collab.net/svn/2008/03/do-not-post-mer.html (which was written before the 1.5 release but is still relevant). You'd do something like this in the working copy of your custom MW code: svn merge \ http://svn.wikimedia.org/svnroot/mediawiki/trunk/@74770 \ http://svn.wikimedia.org/svnroot/mediawiki/trunk/@74798 - Try Piston. I've never tried it myself, but its interface seems nice and I've seen it being recommended on this list before. The dependency on Ruby isn't nice but quite possibly worth it. Good luck, Stefan
Re: Managing modifications to an open source product
On Thu, Oct 14, 2010 at 09:48:21PM +, Dan Nessett wrote: Bad etiquette to answer my own question, but I just reread the Externals Definition section of the book and found this: An externals definition can point only to directories, not to files. So, what I have described won't work even if overloading with svn:externals is supported. You're probably reading an outdated version of the book. See the nightly edition: http://svnbook.red-bean.com/nightly/en/svn.advanced.externals.html File externals are indeed supported as of 1.6, with the caveats I listed in my other reply (those problems aren't documented in the book). The current plan is to fix file externals in the upcoming 1.7 release. But, as I mentioned, externals cannot be used as overlays. Stefan
Re: svn Farm
On Thu, Oct 14, 2010 at 2:22 PM, Stefan Sperling s...@elego.de wrote: I hope the work-in-progress gpg-agent support I mentioned will fill that gap. Would using gpg-agent work for you? Quite likely. I haven't really played with gpg-agent, but I don't know of any reason it shouldn't work. -- David Brodbeck System Administrator, Linguistics University of Washington
Re: Repository organization for complex project
On 14 October 2010 22:03, Les Mikesell lesmikes...@gmail.com wrote:
On 10/14/2010 3:11 PM, David Weintraub wrote:
What? You want a GOOD Maven manual? Real programmers don't use
manuals.
Yeah, I know - they don't write them either (except for subversion, of
course). As you might guess, I'm more of a system administrator than a
programmer...
Fortunately, you don't really have to know too much about Maven if
you're not a Java developer. The intricacies of the pom.xml file don't
concern you. Possibly, the only Maven command you really have to know
is mvn deploy:deploy-file which doesn't require a pom.xml file
http://maven.apache.org/plugins/maven-deploy-plugin/deploy-file-mojo.html.
And, if that command concerns you too much, you can simply revert to
curl.
You do have to understand the Maven repository layout which is a
fairly straight forward hierarchal affair and both Nexus and
Artifactory cover that pretty well.
Thanks - the piece I need is how to map our project tags and
branch/revision_numbers into the maven repo URLs, hopefully in a way that
wouldn't break it for actual maven use since we do have some java developers
here.
every artifact in the maven repository has a set of coordinates, known
as GAV (though technically it is GAVCT)
GroupId:ArtifactId:Version[:Classifier]:Type
in the group id, you use reverse DNS name to establish ownership of
the namespace, dot's are replaced by slashes... so one project I am
working on at github uses the groupId
com.github.stephenc.java-iso-tools as the project url is
http://github.com/stephenc/java-iso-tools
I have other things where I use groupIds starting with com.one-dash
because I own the domain.
The artifactId can be whatever you like (within reason)
The version is a version number
The classifier is similar to artifactId and this is more for
side-artifacts... you probably don't want classifiers in most cases...
side-artifacts would be things like javadocs... try to keep one GAV
for one theing that you build, e.g. a separate GAV for the x86 and the
ARM versions of your build rather than abuse classifiers [so that you
play better with the java builds]
to consturuct the url
URL=${REPO_ROOT}/$(echo ${GROUP_ID} | sed -e
s:\.:/:g)/$ARTIFACT_ID/$VERSION/$ARTIFACT_ID-${CLASSIFIER}-${VERSION}.${TYPE}
if you have a classifier and
URL=${REPO_ROOT}/$(echo ${GROUP_ID} | sed -e
s:\.:/:g)/$ARTIFACT_ID/$VERSION/$ARTIFACT_ID-${VERSION}.${TYPE}
if you don't
-Stephen
--
Les Mikesell
lesmikes...@gmail.com
