Release Repositories
This is not a Subversion question per se. However, as CMs, I hope you guys can help me with what is a CM question. We have a rather interesting setup here at my job. We do some script development, and I'm trying to convince our team to use Subversion for development work. We currently use ClearCase although use is a misnomer. We officially have ClearCase, but it's not available on our Unix development box. About six months ago, someone checked in lot of our scripts into ClearCase, and they haven't been touched since. People are simply making changes to the development scripts without any sort of version control system. Anyway, I am pushing our team to use Subversion, and it'll take a while. Big lumbering beasts don't change directions very quickly. The real issue are vendor patches. We receive patches from a vendor. They're currently stored on a Windows shared drive. When they come in, we get an email from the vendor. We then manually manipulate the patch, then manually FTP it up to a server. We then FTP these patches to a Unix box, run a program that runs a Perl script that fires off several Ant scripts that remunges these files into their final format, and adds in the information into our CM database, so we can track these patches. Oh, by the way, we also have to manually update our ClearQuest database with this information. I've written a script that can take a Vendor patch, reformat it into the final format we need, move it to the server where we want it, and update our CM and ClearQuest databases at the same time -- a completely automated process. What I'd like to do is have a way that when the Vendor puts a patch onto some server, I can trigger my script to process this patch automatically. Subversion's post-commit hook would be a great way to do this. However, Subversion really isn't the tool I need. If we use Subversion, I have to train the vendor in Subversion, and provide them some sort of Subversion client. We also really don't need the version control aspects of Subversion. Patches won't really be changed, and we want to be able to delete obsolete patches in order to free up space on our repository. Subversion, unfortunately, makes that last task very difficult. I've used Nexus and Artifactory as pure release repositories at other sites that have nothing to do with Maven or even Java and Ant. They're easy to integrate with build tools like Hudson and provide a web interface for people to find the software they want and download it. Unfortunately, I don't know if either of these have the same hook mechanism that Subversion has. What I really need is something that is as simple for non-developers to use as Nexus or Artifactory for downloading these patches, but with the power of Subversion's hooks. That way, whenever the vendor downloads a new patch, I can have my script instantly swoop down and process it. Any ideas of any software that can handle this. Again, sorry about bothering the list about what probably is a non-Subversion question. (Then again, Subversion might be exactly what I need if we can overcome some of the other issues I mentioned). However, I know many of you have had similar circumstances and I hope you can shed some light on this issue. -- David Weintraub qazw...@gmail.com
Re: Release Repositories
Greetings, David Weintraub! Any ideas of any software that can handle this. You'd be surprised, but the very filesystem (in Unix/Linux at least) support trigger mechanism. All you need is to write appropriate filter. -- WBR, Andrey Repin (anrdae...@freemail.ru) 22.11.2010, 19:09 Sorry for my terrible english...
RE: Release Repositories
From: Andrey Repin Sent: Monday, November 22, 2010 5:11 PM Any ideas of any software that can handle this. You'd be surprised, but the very filesystem (in Unix/Linux at least) support trigger mechanism. All you need is to write appropriate filter. I think one such mechanism for Linux is called inotify. There should be similar facilities for Windows. Of course, there is a low-level API, but I'm thinking of something that allows you to write: OnCreateFile: call somescript.cmd OnCreateDir: perl someperl.pl The generic name should be *file system event monitor*. But I haven't used any. Michael
Re: Release Repositories
On Mon, Nov 22, 2010 at 11:36, Ludwig, Michael michael.lud...@delphi-mb.de wrote: From: Andrey Repin Sent: Monday, November 22, 2010 5:11 PM Any ideas of any software that can handle this. You'd be surprised, but the very filesystem (in Unix/Linux at least) support trigger mechanism. All you need is to write appropriate filter. I think one such mechanism for Linux is called inotify. There should be similar facilities for Windows. System.IO.FileSystemWatcher in the .NET Framework. http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx FSEvents on MacOS X 10.5 and up. Of course, there is a low-level API, but I'm thinking of something that allows you to write: OnCreateFile: call somescript.cmd OnCreateDir: perl someperl.pl The generic name should be *file system event monitor*. But I haven't used any. Michael
Re: Release Repositories
On 11/22/2010 10:44 AM, Andy Levy wrote: On Mon, Nov 22, 2010 at 11:36, Ludwig, Michael michael.lud...@delphi-mb.de wrote: From: Andrey Repin Sent: Monday, November 22, 2010 5:11 PM Any ideas of any software that can handle this. You'd be surprised, but the very filesystem (in Unix/Linux at least) support trigger mechanism. All you need is to write appropriate filter. I think one such mechanism for Linux is called inotify. There should be similar facilities for Windows. System.IO.FileSystemWatcher in the .NET Framework. http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx FSEvents on MacOS X 10.5 and up. Of course, there is a low-level API, but I'm thinking of something that allows you to write: OnCreateFile: call somescript.cmd OnCreateDir: perl someperl.pl The generic name should be *file system event monitor*. But I haven't used any. Realistically, you probably don't need to kick off the job the instant the filesystem changes - you'll at least want to wait until the file transfer completes. I'd expect a scheduled job running from cron on linux or the windows task scheduler checking for new files every few minutes to work at least as well as your existing manual process and avoid any OS/filesystem dependencies (i.e. it could run from linux checking a smb/cifs mounted windows filesystem or the reverse if you want). -- Les Mikesell lesmikes...@gmail.com
Re: Release Repositories
Well, the file system triggers are nice to know, but I'm too low level a peon in order to implement such a policy on our servers. Besides, we use AIX and not Linux here. So I have the following choices: * Use a crontab entry to look for changes every five minutes or so. * Use Hudson and use its File System CM plugin to do the same thing. * Both Nexus and Artifactory can use plugins written in Groovy. Unfortunately, I don't know Groovy, and the documentation is rather light on details. * Use Subversion the triggers do what I want, but training a vendor to use it might be too difficult. You have to checkout the directory, do a svn add', and then do a svn commit. I'm having a hard enough time getting the developers here to understand that. I'm leaning towards using Artifactory or Nexus as the actual release managers for a variety of reasons, and then using Hudson's ability to examine the file system for changes in the directory where the downloaded patches would be stored. I want to get away from granting direct access to vendors on our servers, and I both Nexus and Artifactory will have an interface that's pretty simple to use. Using Hudson allows me to monitor the directories without asking for scheduling abilities to run a process every five minutes. The powers that be will probably not mind Hudson too much, but hate crontab stuff. Seems a little too Rube Goldberg for my tastes, but it seems like the simplest and easiest thing to document. Hudson gives us some reporting and emailing capabilities built into the system. -- David Weintraub qazw...@gmail.com
Re: Release Repositories
On 11/22/2010 11:24 AM, David Weintraub wrote: Well, the file system triggers are nice to know, but I'm too low level a peon in order to implement such a policy on our servers. Besides, we use AIX and not Linux here. So I have the following choices: * Use a crontab entry to look for changes every five minutes or so. * Use Hudson and use its File System CM plugin to do the same thing. * Both Nexus and Artifactory can use plugins written in Groovy. Unfortunately, I don't know Groovy, and the documentation is rather light on details. * Use Subversion the triggers do what I want, but training a vendor to use it might be too difficult. You have to checkout the directory, do a svn add', and then do a svn commit. I'm having a hard enough time getting the developers here to understand that. I'm leaning towards using Artifactory or Nexus as the actual release managers for a variety of reasons, and then using Hudson's ability to examine the file system for changes in the directory where the downloaded patches would be stored. I want to get away from granting direct access to vendors on our servers, and I both Nexus and Artifactory will have an interface that's pretty simple to use. Using Hudson allows me to monitor the directories without asking for scheduling abilities to run a process every five minutes. The powers that be will probably not mind Hudson too much, but hate crontab stuff. I'm not that familiar with AIX, but linux has per-user crontabs so if you have a login and access to the files under that login you would automatically have scheduling abilities. I thought that was true for most things based on SysV's cron. Seems a little too Rube Goldberg for my tastes, but it seems like the simplest and easiest thing to document. Hudson gives us some reporting and emailing capabilities built into the system. Assuming the stock mail system works, cron will email you the output of its jobs - or you can redirect the output to a file if you want. Hudson may give you a more familiar interface but it is basically going to be a scheduler for the jobs you write anyway, not adding any capabilities you would not have at the unix command line in the language of your choice. -- Les Mikesell lesmikes...@gmail.com
Re: Release Repositories
On Mon, Nov 22, 2010 at 12:50 PM, Les Mikesell lesmikes...@gmail.com wrote: I'm not that familiar with AIX, but linux has per-user crontabs so if you have a login and access to the files under that login you would automatically have scheduling abilities. I thought that was true for most things based on SysV's cron. It is true if the System Administrator permits it. This is configurable on a per user basis -- either the user is in cron.allow or cron.deny. If neither of these files exist, it is most likely you don't have cron permissions. Our System Admin doesn't allow it. -- David Weintraub qazw...@gmail.com
Re: Release Repositories
On 11/22/2010 12:14 PM, David Weintraub wrote: On Mon, Nov 22, 2010 at 12:50 PM, Les Mikeselllesmikes...@gmail.com wrote: I'm not that familiar with AIX, but linux has per-user crontabs so if you have a login and access to the files under that login you would automatically have scheduling abilities. I thought that was true for most things based on SysV's cron. It is true if the System Administrator permits it. This is configurable on a per user basis -- either the user is in cron.allow or cron.deny. If neither of these files exist, it is most likely you don't have cron permissions. Our System Admin doesn't allow it. Make him do all the work then, if that's his choice. -- Les Mikesell lesmikes...@gmail.com
Re: Release Repositories
Greetings, Les Mikesell! Realistically, you probably don't need to kick off the job the instant the filesystem changes - you'll at least want to wait until the file transfer completes. I'd expect a scheduled job running from cron on linux or the windows task scheduler checking for new files every few minutes to work at least as well as your existing manual process and avoid any OS/filesystem dependencies (i.e. it could run from linux checking a smb/cifs mounted windows filesystem or the reverse if you want). Cron job won't be able to know if file transfer is completed. It will have to guess from, e.g., testing the archive (if it's archive) for integrity. filesystem notification mechanism will for sure know, when the iostream got filewriteclose event. -- WBR, Andrey Repin (anrdae...@freemail.ru) 23.11.2010, 5:53 Sorry for my terrible english...
Re: Release Repositories
On 11/22/10 8:55 PM, Andrey Repin wrote: Greetings, Les Mikesell! Realistically, you probably don't need to kick off the job the instant the filesystem changes - you'll at least want to wait until the file transfer completes. I'd expect a scheduled job running from cron on linux or the windows task scheduler checking for new files every few minutes to work at least as well as your existing manual process and avoid any OS/filesystem dependencies (i.e. it could run from linux checking a smb/cifs mounted windows filesystem or the reverse if you want). Cron job won't be able to know if file transfer is completed. It will have to guess from, e.g., testing the archive (if it's archive) for integrity. filesystem notification mechanism will for sure know, when the iostream got filewriteclose event. You can wait until the file timestamp stops changing for a while - but neither that nor a filesystem notification will tell you if the transfer actually completed or if it failed midway. However, many transfers use a temporary name, renaming only when successful and complete so just looking for an expected name may be enough. -- Les Mikesell lesmikes...@gmail.com
Re: Release Repositories
On Mon, Nov 22, 2010 at 9:55 PM, Andrey Repin anrdae...@freemail.ru wrote: Greetings, Les Mikesell! Realistically, you probably don't need to kick off the job the instant the filesystem changes - you'll at least want to wait until the file transfer completes. I'd expect a scheduled job running from cron on linux or the windows task scheduler checking for new files every few minutes to work at least as well as your existing manual process and avoid any OS/filesystem dependencies (i.e. it could run from linux checking a smb/cifs mounted windows filesystem or the reverse if you want). Cron job won't be able to know if file transfer is completed. It will have to guess from, e.g., testing the archive (if it's archive) for integrity. filesystem notification mechanism will for sure know, when the iostream got filewriteclose event. One of the more helpful tricks is to check if it's newer than a certain period, and *older* than a certtain, modest period (such as a few minutes) to make sure the transfer is complete.
