Re: How to Keep Track of Changes to the System
On Friday, September 01, 2017 01:40:33 AM hdv@gmail wrote: > On 2017-08-31 23:36, rhkra...@gmail.com wrote: > > I'm interested as well, if you could send a copy to me. > > > > @Ray: And even more so, Ray, after you modify it. > I sent it to this list yesterday. If you missed it I can send it again. > Just let me know. Thanks, I saw it and saved it! I also looked at it briefly (i.e., skimmed)--Perl is not my thing (if anything is) so, at some point, I will have to spend some time understanding how to use it--I suppose at the very least, put it in a file and mark it executable, and then invoke it with the file name. I don't expect to have trouble with that, but also am busy with some other things for the next couple weeks, so not sure when I'll get to that--when I do, if I have questions, I have your email address. Thanks again, Randy Kramer
Re: How to Keep Track of Changes to the System
On 2017-08-31 23:36, rhkra...@gmail.com wrote: > On Wednesday, August 30, 2017 10:39:38 PM ray wrote: >> ...snip >> >> Yes, I would like to work with this. I should be able to modify the perl >> script to also save a tag file to hold the metadata. So now I have a >> reason to learn some perl. > > I'm interested as well, if you could send a copy to me. > > @Ray: And even more so, Ray, after you modify it. > Hi rhkramer, I sent it to this list yesterday. If you missed it I can send it again. Just let me know. Grx HdV
Re: How to Keep Track of Changes to the System
On Wednesday, August 30, 2017 10:56:44 PM ray wrote: > Thank you. This is an interesting way to store the broken system. It will > be like a junk yard that I can copy out of. > > Partitioning will be a challenge. Currently, this is laptop runs LVM. I > have two groups. The system is on one, the other I plan on using for VM > storage. I could break the one for VMs into groups for file systems. I > am not sure how to use this. Once I have these different groups, if I > want to rebuild, I will need to have the new system installed to one of > the unused VGs. I have a quandary about this each time I rebuild, I have > a challenge with the Debian installer on where to put the new system. It > seem like I rebuild the LVM each time which means I would wipe out the > previous system. You're welcome! I don't / have never used LVM, but I thought that at least in some scenarios, you created partitions and then installed LVM somehow on top of those partitions. I assume that, if you create one more partition, you could just leave it out of LVM? Maybe someone else can advise you.
Re: How to Keep Track of Changes to the System
On Wednesday, August 30, 2017 10:39:38 PM ray wrote: > ...snip > > Yes, I would like to work with this. I should be able to modify the perl > script to also save a tag file to hold the metadata. So now I have a > reason to learn some perl. I'm interested as well, if you could send a copy to me. @Ray: And even more so, Ray, after you modify it.
Re: How to Keep Track of Changes to the System
On 2017-08-31 04:39, ray wrote: > On Sunday, August 27, 2017 at 6:50:06 AM UTC-5, hdv@gmail wrote: >> On 2017-08-26 05:14, ray wrote: >>> I would like to find a way to keep track of changes I make to my system. >>> ...snip >> Hi Ray, >> >> I just returned from a short holiday, so I am a bit late to the party, >> but... if >> you don't want to set up a full versioning system I might have something else >> for you. About 10 years ago I had the same need as you. What I did was write >> a >> perl-script that automatically makes a timestamped backup of each file you >> edit >> to a directory you define yourself (in that directory the full path of the >> original is preserved). You use it like visudo, you just call it like this: >> >> vicf >> >> All the rest happens automagically. >> >> Of course this will only help for plain-text files and it doesn't provide for >> the annotations you mentioned. But if you are interested I >> can mail it to you. >> >> Grx HdV > ...snip > > Yes, I would like to work with this. I should be able to modify the perl > script to also save a tag file to hold the metadata. So now I have a reason > to learn some perl. > > Thank you. > Ray Here it is. Make sure to set the variables in the section "User-definable defaults". I use the SWITCH statement to configure the script for use on multiple systems. You might not need that. In that case just make sure you set $root. Good luck with it! If you make any changes, please share them with the list. Maybe others might find them useful too. Grx HdV ===[begin script] #!/usr/bin/perl #TODO : add support for settings stored in ~/.vicfrc or an explicitly given file our $VERSION = '0.92'; use strict; use warnings; use Getopt::Long qw(:config bundling); use Pod::Usage; use Sys::Hostname; use Cwd qw(realpath getcwd); use POSIX qw(strftime sysconf _PC_CHOWN_RESTRICTED); use File::Spec; use File::Copy; my $hostname = hostname(); #User-definable defaults my $root = ''; SWITCH: { if ($hostname eq '') { $root = ''; last SWITCH; } if ($hostname eq '') { $root = ''; last SWITCH; } if ($hostname eq '') { $root = ''; last SWITCH; } } my $datetime_format = '-%Y%m%d'; #d my $sequencenr_format = '-%02d'; #n my $append_sequencenr = 1; #a my $keep_permissions = 1; #p my $keep_owner = 1;#o my $keep_group = 1;#g my $keep_times = 1;#t my $backup_file = 0; #b my $x_editor = 0; #x my $editor_path = '/usr/bin/vim'; my $x_editor_path = '/usr/bin/gvim'; my $backup_option = '-c "set backup"'; my $no_backup_option = '-c "set nobackup"'; #Internal variables #Defaults for commandline options my $help = 0; my $manual = 0; my $show_version = 0; my $debug = 0; my $verbose = 0; #Parse the commandline arguments #Get all options GetOptions(#Standard options 'debug|D+', \$debug, 'help!', \$help, 'h|?', \$help, 'manual!', \$manual, 'version!', \$show_version, 'V', \$show_version, 'verbose|v+', \$verbose, #Options specific for this program 'root|r=s', \$root, 'datetime|d=s', \$datetime_format, 'sequencenr|n=s', \$sequencenr_format, 'append_sequencenr!', \$append_sequencenr, 'a!', \$append_sequencenr, 'permissions!', \$keep_permissions, 'p!', \$keep_permissions, 'owner!', \$keep_owner, 'o!', \$keep_owner, 'group!', \$keep_group, 'g!', \$keep_group, 'times!', \$keep_times, 't!', \$keep_times, 'backup!', \$backup_file, 'b!', \$backup_file, 'x_editor!', \$x_editor, 'x!', \$x_editor ) or pod2usage(0); pod2usage(verbose => 1, exitval => 0) if $help; pod2usage(verbose => 2, exitval => 0) if $manual; if ($show_version) { print "vicf version $VERSION (c) 2015 Jadev\n"; exit 0; } #Assign the first non-option argument to a variable for easier use die "No file to be edited was given.\n" unless $ARGV[0]; my $source = $ARGV[0]; #Sanitize paths for easier use $root = expand_path($root); die "Path pointing to the repository ($root) is invalid.\n" unless $root; $source = expand_path($source); die "Path pointing to the file to be edi
Re: How to Keep Track of Changes to the System
On Sunday, August 27, 2017 at 9:00:05 AM UTC-5, rhkr...@gmail.com wrote: ...snip... Hi RHKR, Thank you. This is an interesting way to store the broken system. It will be like a junk yard that I can copy out of. Partitioning will be a challenge. Currently, this is laptop runs LVM. I have two groups. The system is on one, the other I plan on using for VM storage. I could break the one for VMs into groups for file systems. I am not sure how to use this. Once I have these different groups, if I want to rebuild, I will need to have the new system installed to one of the unused VGs. I have a quandary about this each time I rebuild, I have a challenge with the Debian installer on where to put the new system. It seem like I rebuild the LVM each time which means I would wipe out the previous system. Ray Ray
Re: How to Keep Track of Changes to the System
On Sunday, August 27, 2017 at 6:50:06 AM UTC-5, hdv@gmail wrote: > On 2017-08-26 05:14, ray wrote: > > I would like to find a way to keep track of changes I make to my system. > > ...snip > Hi Ray, > > I just returned from a short holiday, so I am a bit late to the party, but... > if > you don't want to set up a full versioning system I might have something else > for you. About 10 years ago I had the same need as you. What I did was write a > perl-script that automatically makes a timestamped backup of each file you > edit > to a directory you define yourself (in that directory the full path of the > original is preserved). You use it like visudo, you just call it like this: > > vicf > > All the rest happens automagically. > > Of course this will only help for plain-text files and it doesn't provide for > the annotations you mentioned. But if you are interested I > can mail it to you. > > Grx HdV ...snip Yes, I would like to work with this. I should be able to modify the perl script to also save a tag file to hold the metadata. So now I have a reason to learn some perl. Thank you. Ray
Re: How to Keep Track of Changes to the System
On 25/08/17 22:14, ray wrote: > I would like to find a way to keep track of changes I make to my system. It > seem that I may learn from others on how they keep track of changes they make > to their systems. I have a plain-text file of notes, which I keep under Mercurial version control. I make a note here whenever I make any big change. For manually installed packages, I install them under a directory in “~/local/stow”. For example “~/local/stow/emacs”. Since there is a one-to-one correspondence between packages and directories under the “stow” directory, obtaining a list of what packages I have installed is as easy as “ls ~/local/stow”. The search path for executables is “~/local/bin”. I use GNU Stow to automatically make symbolic links from here to the corresponding directory under “~/local/stow”. I can recommend GNU Stow to have better control over *manually* installed packages. A common problem is that ones does “make install” and then when one wants to delete the package, one does not know what files one should delete, and ones does not realize if something is being overwritten. GNU Stow solves that. -- Do not eat animals, respect them as you respect people. https://duckduckgo.com/?q=how+to+(become+OR+eat)+vegan signature.asc Description: OpenPGP digital signature
Re: How to Keep Track of Changes to the System
On 26/08/17 16:10, Joe wrote: >> Thank you for the list of solutions. It is interesting that SVN can >> be used with etckeeper. It looks like I should learn git. I have >> used SVN for other things, but I am easily pulled from my comfort >> zone for value. > > Git is very widely used, and on important projects, so it is being > vigorously maintained. It's probably the right choice for new projects. New project should use Mercurial. Existing project should switch to Mercurial. Although Git can do anything that can be needed (and so can Mercurial), the difference is that Git has an horrendously designed interface and the concepts it is based on are many times irrational. For example, consider the “git reset” command. This one deserves an award for the most irrationally designed command in all of GNU/Linux. If you want to change what commit the current branch points to, you must use “git reset”, and you must memorize the meanings of “--soft”, “--mixed”, “--hard” and “--keep”. If you do it wrong (e.g.: “keep” instead of “--mixed”, you lose data). But surprisingly, git reset does not always change what commit the current branch points to. Sometimes it just moves files from a commit to the index (“staging area”). (“git reset -- .”). So “git reset” does 4 things with little relation: Sometimes it moves what commit the current branch points to and *maybe* changes the working directory and staging area. Sometimes it just de-stages your staged changes (“git reset”). Sometimes it cancels a failed merge. Sometimes it copies files from an older commit to the staging area. These tasks are group into a single command for no logical reason. Now, this is no problem *after* you have learned how to use Git. You may even think that well, it ought to be difficult to learn how to use it, but that is incorrect. It is difficult only because the interface and the concepts behind it are irrationally designed. By contrast, Mercurial interface is very clean. Every command always does one simple thing. This does not mean that it is less powerful. Although in a previous time (years ago) when one could not do powerful history modification as in Git, that era is now history. Imagine how many man-hours are wasted learning how to use Git, which would otherwise be used to do actual programming. -- Do not eat animals, respect them as you respect people. https://duckduckgo.com/?q=how+to+(become+OR+eat)+vegan signature.asc Description: OpenPGP digital signature
Re: How to Keep Track of Changes to the System
On Friday, August 25, 2017 11:14:52 PM ray wrote: > I would like to find a way to keep track of changes I make to my system. > It seem that I may learn from others on how they keep track of changes > they make to their systems. > > When I make changes, I don't remember where I made changes or why. > > It would be great to have a log of what changes I've made, where they were > made, how they were made (direct edit, scripted, etc.), why I made them, > references that I used to determine the change, and what was the outcome. > > Right now, I get lost in my documentation. I research solutions, make > notes in Onenote on a Windows machine, record configurations files that I > will test. But It is difficult to record results such as syslogs or > console transactions. More challenging is that I have different notebook > tabs for different objectives. So when I want to see what I changed, I > have to go through many different objectives because I don't know what > object I was shooting for when I made the change. > > I would really like to hear how others track their changes or suggestions > how I may tack changes. > > I store all the changes on a different computer because I screw up the > installation on my machine under test and rebuild the OS. The laptop I am > building to run Xen is on its 28th build. > > I would appreciate any suggestions. In general, I have similar problems, and, I don't have a good general solution or suggestion at this time, but I do have one suggestion to address one specific part of the problem. I would suggest that you create a separate partition on the "subject" computer (the one on which you rebuild the OS when you have a (significant) problem with the OS. Then: * mount that partition each (or almost each) time you rebuild the OS * never reformat or delete that partition * when you mount the partition, create a mounting point outside the FHS hierarchy (i.e., not in /home)--I typically create several such partitions (for different reasons) using my initials or similar, for instance, I have rhk01, rhk02, ..., back01, back02 and such * keep the change log (whatever form it takes on such a partition--with luck they will always be available, and, it is easier to c&p to a file on the computer in question rather than "transcribe" stuff to a different computer. Of course, a similar result could be obtained by mounting a partion on some other computer over the network (using, for example, NFS or Samba), or using SSL to write notes on the subject computer but store them on a "safer" computer. Of course, in any case, the files on that separate partition should be backed up regulalrly, just in case (and, particularly, just before you start an OS rebuild). With respect to an actual general change log (as opposed to where to keep it), I keep my notes in some specially formatted files for which I've created some "programming" (e.g., macros, and similar) to create something that I call (at various times): my askSam workalike, my offline wiki-like thing, and some other similar names that I don't recall at the moment. (Oh, I also consider it a "mashup" as it (each iteration) uses a variety of programs to provide all the features I want to have (for example, recoll for full text indexed search). I've gone through several iterations, each using an editor with special macros to make things more convenient, but I don't currently have any features that do anything like monitor all of the files in /etc (and/or /var/log or similar locations) and record any changes, the date, and prompt me to enter a reason for the change. That would be nice! (Previous iterations have been based on nedit as the editor, then kate as the editor (with a different feature set, partially based on the capabilties of the editor), and have also included a full text indexed search system (using recoll). My current focus (which I've been "working on" sporadically over the last 12 years or so, will be based on scintilla, which because scintilla is used as the editing "widgit" for quite a few editors, would open the door to letting me choose from a variety of editors. (And then, maybe (not something I really considered before today), might include something like monitoring /home, etc. for changes (as mentioned somewhere above) (using things like-- well, I can't remember the name, but there are tools that monitor files for changes, and scintilla-based editors have a pretty powerful scripting language in Lua.
Re: How to Keep Track of Changes to the System
On 2017-08-26 05:14, ray wrote: > I would like to find a way to keep track of changes I make to my system. It > seem that I may learn from others on how they keep track of changes they make > to their systems. > > When I make changes, I don't remember where I made changes or why. > > It would be great to have a log of what changes I've made, where they were > made, how they were made (direct edit, scripted, etc.), why I made them, > references that I used to determine the change, and what was the outcome. > > Right now, I get lost in my documentation. I research solutions, make notes > in Onenote on a Windows machine, record configurations files that I will > test. But It is difficult to record results such as syslogs or console > transactions. More challenging is that I have different notebook tabs for > different objectives. So when I want to see what I changed, I have to go > through many different objectives because I don't know what object I was > shooting for when I made the change. > > I would really like to hear how others track their changes or suggestions how > I may tack changes. > > I store all the changes on a different computer because I screw up the > installation on my machine under test and rebuild the OS. The laptop I am > building to run Xen is on its 28th build. > > I would appreciate any suggestions. > > Ray > Hi Ray, I just returned from a short holiday, so I am a bit late to the party, but... if you don't want to set up a full versioning system I might have something else for you. About 10 years ago I had the same need as you. What I did was write a perl-script that automatically makes a timestamped backup of each file you edit to a directory you define yourself (in that directory the full path of the original is preserved). You use it like visudo, you just call it like this: vicf All the rest happens automagically. Of course this will only help for plain-text files and it doesn't provide for the annotations you mentioned. But if you are interested I can mail it to you. Grx HdV P.S. Here's the output of the help so you can decide if this is what you need: vicf --help Usage: vicf [options] List of options: [-h|--help|-?] [--manual] [-V|--version] [-r|--root ] [-d|--datetime ] [-n|--sequencenr ] [-a|--append_sequencenr] [-p|--permissions] [-o|--owner] [-g|--group] [-t|--times] [-b|--backup] [-x|--x_editor] Options: --help Print a brief help message. --manual Print the full manual. --version Print version and copyright information. --root The directory under which a dated copy of the original file should be stored. --datetime A format string suitable for strftime(). Together with the local time this parameter will used as input for strftime(). The result will be appended to the name of the target file. See man 3 strftime for more details. --sequencenr A format string suitable for sprintf(). This will be used to generate a sequence number, which will be append to the filename if the target file already exists. --append_sequencenr Append a sequence number to the filename, even if it does not exist. Used to start sequences at 1 instead of 2. --permissions Preserve the access permissions of the original file. --owner Preserve the owner of the original file. --group Preserve the group of the original file. --times Preserve the access and modification times of the original file. --backup Instruct the editor to make a backup of the original file. This is a convenience option that has nothing to do with the dated copy of the original file. --x_editor Start the editor in graphical mode instead of console mode. At the top of the code there is a section named 'User-definable defaults' where defaults appropriate for the current environment can be set. Doing so alleviates the need to specify options on every invocation of the program. Arguments: This program accepts only one argument, which is the path to the file to be edited.
Re: How to Keep Track of Changes to the System
On Sat, 26 Aug 2017 06:02:07 -0700 (PDT) ray wrote: > > Thank you for the list of solutions. It is interesting that SVN can > be used with etckeeper. It looks like I should learn git. I have > used SVN for other things, but I am easily pulled from my comfort > zone for value. Git is very widely used, and on important projects, so it is being vigorously maintained. It's probably the right choice for new projects. > > There is an interesting challenge here on where/how to keep > repositories on a laptop. It is valuable to have them locally as > often my problems are networking; if the repositories are local, I > can use another box to view them, but sometimes it may be a challenge > to move files when connectivity is lost. I am sure there is an > architecture that will be suitable. Git creates a repository (by default) within the directory you base it on, so copying the directory copies the repository. Git, both command-line and GUI, exists for *nix and Windows, and a repository can be operated from either. A backup of /etc to a USB stick, containing a git repository, can be opened by git on another machine, and another platform. I actually use git mostly on Windows, and mostly for its 'intended' use with software, though I also use it to track circuit diagrams and PCB layouts under construction. -- Joe
Re: How to Keep Track of Changes to the System
On Sat, Aug 26, 2017 at 06:02:07AM -0700, ray wrote: > On Saturday, August 26, 2017 at 5:10:05 AM UTC-5, Dan Ritter wrote: > ... > > For a single system, etckeeper is an excellent choice. It stores > > changes anywhere in /etc (and optionally in other places) in > > git or subversion. > > > > You can go a little farther easily by learning git directly. > > https://try.github.io/levels/1/challenges/1 is a good resource. > > > > If you have multiple systems that need to be handled in similar > > ways, you want to learn an advanced configuration management > > system like chef, puppet, ansible, salt... > > > > -dsr- > > Dan, > > Thank you for the list of solutions. It is interesting that SVN can be used > with etckeeper. It looks like I should learn git. I have used SVN for other > things, but I am easily pulled from my comfort zone for value. > > There is an interesting challenge here on where/how to keep repositories on a > laptop. It is valuable to have them locally as often my problems are > networking; if the repositories are local, I can use another box to view > them, but sometimes it may be a challenge to move files when connectivity is > lost. I am sure there is an architecture that will be suitable. Sure. git is distributed; you can easily check it out on another system, and thus have backups anytime you are connected to a network, while having a complete local record as well. > It seems like there may be value in using both a config mgr and a version control system. I will start checking into these to better understand. They are complementary solutions; config management solves "how to get things in the right shape" problems, and version control solves "what did this look like before?" problems. -dsr-
Re: How to Keep Track of Changes to the System
On Saturday, August 26, 2017 at 5:10:05 AM UTC-5, Dan Ritter wrote: ... > For a single system, etckeeper is an excellent choice. It stores > changes anywhere in /etc (and optionally in other places) in > git or subversion. > > You can go a little farther easily by learning git directly. > https://try.github.io/levels/1/challenges/1 is a good resource. > > If you have multiple systems that need to be handled in similar > ways, you want to learn an advanced configuration management > system like chef, puppet, ansible, salt... > > -dsr- Dan, Thank you for the list of solutions. It is interesting that SVN can be used with etckeeper. It looks like I should learn git. I have used SVN for other things, but I am easily pulled from my comfort zone for value. There is an interesting challenge here on where/how to keep repositories on a laptop. It is valuable to have them locally as often my problems are networking; if the repositories are local, I can use another box to view them, but sometimes it may be a challenge to move files when connectivity is lost. I am sure there is an architecture that will be suitable. It seems like there may be value in using both a config mgr and a version control system. I will start checking into these to better understand. Thank you, Ray
Re: How to Keep Track of Changes to the System
On Saturday, August 26, 2017 at 4:50:06 AM UTC-5, Nemeth Gyorgy wrote: ... > I use etckeeper. Of course in tracks changes in /etc (and > subdirectories) only, but it is enough for me. Nemeth, Thank you. I like the idea of using a version control system. Ray
Re: How to Keep Track of Changes to the System
On Friday, August 25, 2017 at 11:10:04 PM UTC-5, Andy Smith wrote: > Hi, > ...> > I won't always go as far as installing and configuring a package > with the config management straight off. Depending on what system I > am working on I will sometimes "cheat" and install it manually with > apt, configure the files with an editor etc. But I do always at > least try to "go back" and recreate the working config with > config management so it's repeatable. Andy, Thank you. I did not know these existed. I am going to study these opportunities. Ray
Re: How to Keep Track of Changes to the System
On Fri, Aug 25, 2017 at 08:14:52PM -0700, ray wrote: > I would like to find a way to keep track of changes I make to my system. It > seem that I may learn from others on how they keep track of changes they make > to their systems. > > When I make changes, I don't remember where I made changes or why. > > It would be great to have a log of what changes I've made, where they were > made, how they were made (direct edit, scripted, etc.), why I made them, > references that I used to determine the change, and what was the outcome. For a single system, etckeeper is an excellent choice. It stores changes anywhere in /etc (and optionally in other places) in git or subversion. You can go a little farther easily by learning git directly. https://try.github.io/levels/1/challenges/1 is a good resource. If you have multiple systems that need to be handled in similar ways, you want to learn an advanced configuration management system like chef, puppet, ansible, salt... -dsr-
Re: How to Keep Track of Changes to the System
2017-08-26 05:14 keltezéssel, ray írta: > I would like to find a way to keep track of changes I make to my system. It > seem that I may learn from others on how they keep track of changes they make > to their systems. > > When I make changes, I don't remember where I made changes or why. > > It would be great to have a log of what changes I've made, where they were > made, how they were made (direct edit, scripted, etc.), why I made them, > references that I used to determine the change, and what was the outcome. > > Right now, I get lost in my documentation. I research solutions, make notes > in Onenote on a Windows machine, record configurations files that I will > test. But It is difficult to record results such as syslogs or console > transactions. More challenging is that I have different notebook tabs for > different objectives. So when I want to see what I changed, I have to go > through many different objectives because I don't know what object I was > shooting for when I made the change. > > I would really like to hear how others track their changes or suggestions how > I may tack changes. > > I store all the changes on a different computer because I screw up the > installation on my machine under test and rebuild the OS. The laptop I am > building to run Xen is on its 28th build. > > I would appreciate any suggestions. I use etckeeper. Of course in tracks changes in /etc (and subdirectories) only, but it is enough for me.
Re: How to Keep Track of Changes to the System
Hi, On Fri, Aug 25, 2017 at 08:14:52PM -0700, ray wrote: > I would really like to hear how others track their changes or suggestions how > I may tack changes. I configure almost everything with configuration management like puppet or ansible. Then the configuration is treated like code, can be documented like code, stored in revision management (e.g. git) like code. I honestly don't think it's overkill for even one system, because as you say, it is tricky to document things properly even for one system. I won't always go as far as installing and configuring a package with the config management straight off. Depending on what system I am working on I will sometimes "cheat" and install it manually with apt, configure the files with an editor etc. But I do always at least try to "go back" and recreate the working config with config management so it's repeatable. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting