Re: hammer mirror shows differrent results from differrent terminals
:pkill hammer :hammer synctid /Backup1/Data quick :hammer mirror-copy /Backup1/Data /Backup2/Data : :at the end of /etc/rc.shutdown : :the "pkill hammer" command kills all "mirror-stream" processes but :does not kill any kernel helper threads. Is that OK? : :also while starting "hammer mirror-stream" using /etc/rc.local there :are 3 mirror-stream processes. : :root574 0.0 0.0 396 136 con- IL8:31AM 0:00.00 hammer :mirror-stream /Backup1/Data /Backup2/Data :root591 0.0 0.0 1100 236 con- IL8:31AM 0:00.00 hammer :mirror-stream /Backup1/Data /Backup2/Data :root592 0.0 0.0 1100 224 con- IL8:31AM 0:00.00 hammer :mirror-stream /Backup1/Data /Backup2/Data : :Is this OK? I have a doubt since you said " The mirroring utility is :not designed to handle : parallel feeds to the same PFS target." : :Thanks : :Siju It may not die immediately when you kill it. You definitely want to use the lockf(1) utility to ensure that you are only running one thing at a time (as appropriate for what you are doing). hammer mirror-stream forks itself twice so it can exec ssh. It should exec ssh via those forks though I'm not sure what actually shows up in the ps. -Matt Matthew Dillon
Re: hammer mirror shows differrent results from differrent terminals
On Thu, Jul 16, 2009 at 8:16 AM, Matthew Dillon wrote: > :hammer synctid /Backup1/Data quick > :hammer mirror-copy /Backup1/Data /Backup2/Data > : > :Or should i kill all the "mirror-stream" processes > > You should kill all the mirror-stream processes before doing a > mirror-copy. The mirroring utility is not designed to handle > parallel feeds to the same PFS target. > OK I use the lines pkill hammer hammer synctid /Backup1/Data quick hammer mirror-copy /Backup1/Data /Backup2/Data at the end of /etc/rc.shutdown the "pkill hammer" command kills all "mirror-stream" processes but does not kill any kernel helper threads. Is that OK? also while starting "hammer mirror-stream" using /etc/rc.local there are 3 mirror-stream processes. root574 0.0 0.0 396 136 con- IL8:31AM 0:00.00 hammer mirror-stream /Backup1/Data /Backup2/Data root591 0.0 0.0 1100 236 con- IL8:31AM 0:00.00 hammer mirror-stream /Backup1/Data /Backup2/Data root592 0.0 0.0 1100 224 con- IL8:31AM 0:00.00 hammer mirror-stream /Backup1/Data /Backup2/Data Is this OK? I have a doubt since you said " The mirroring utility is not designed to handle parallel feeds to the same PFS target." Thanks Siju
Re: hammer mirror shows differrent results from differrent terminals
:hammer synctid /Backup1/Data quick :hammer mirror-copy /Backup1/Data /Backup2/Data : :Or should i kill all the "mirror-stream" processes You should kill all the mirror-stream processes before doing a mirror-copy. The mirroring utility is not designed to handle parallel feeds to the same PFS target. :root -1 0.0 0.0 00 ?? BL 12:14PM 0:00.00 (hammer-S3) :root -1 0.0 0.0 00 ?? BL 12:14PM 0:00.00 (hammer-S2) :root -1 0.0 0.0 00 ?? BL 12:14PM 0:00.00 (hammer-S1) :root -1 0.0 0.0 00 ?? BL 12:14PM 0:00.00 (hammer-S0) :root -1 0.0 0.0 00 ?? BL 12:14PM 0:00.00 (hammer-M) : :before giving the : :"hammer mirror-copy" command in /etc/rc.shutdown ?? : :also why are there 3 hammer mirror-stream processes? :and what are the hammer-M, hammer-S* processes? : :Thankyou so much : :--Siju The hammer-M/S* processes are kernel helper threads for HAMMER. They are responsible for fsync() and flush sequencing. -Matt Matthew Dillon
Re: hammer mirror shows differrent results from differrent terminals
On Wed, Jul 8, 2009 at 9:31 PM, Matthew Dillon wrote: > > Mirror-stream works on HAMMER's B-Tree elements. Whenever a HAMMER > flush occurs (every 30-60 seconds automatically or if you sync) any > B-Tree elements which are deleted, updated, or inserted are sent across > the wire to the slave. > Thankyou so much For you reply Matt :-) I currently start mirroing with this line in /etc/rc.local hammer mirror-stream /Backup1/Data /Backup2/Data So Inorder to make sure that the master PFS and Slave PFS are in sync when a shut down happens is it enough to put these lines in /etc/rc.shutdown ? hammer synctid /Backup1/Data quick hammer mirror-copy /Backup1/Data /Backup2/Data Or should i kill all the "mirror-stream" processes root592 60.1 0.1 1036 732 con- RL 12:14PM 0:03.58 hammer mirror-stream /Backup1/Data /Backup2/Data root591 41.1 0.1 1036 744 con- SL 12:14PM 0:02.69 hammer mirror-stream /Backup1/Data /Backup2/Data root -1 0.0 0.0 00 ?? BL 12:14PM 0:00.00 (hammer-S1) root -1 0.0 0.0 00 ?? BL 12:14PM 0:00.00 (hammer-S0) root -1 0.0 0.0 00 ?? BL 12:14PM 0:00.00 (hammer-M) root -1 0.0 0.0 00 ?? BL 12:14PM 0:00.00 (hammer-S3) root -1 0.0 0.0 00 ?? BL 12:14PM 0:00.00 (hammer-S2) root -1 0.0 0.0 00 ?? BL 12:14PM 0:00.00 (hammer-S1) root -1 0.0 0.0 00 ?? BL 12:14PM 0:00.00 (hammer-S0) root -1 0.0 0.0 00 ?? BL 12:14PM 0:00.00 (hammer-M) root574 0.0 0.0 396 136 con- IL 12:14PM 0:00.00 hammer mirror-stream /Backup1/Data /Backup2/Data root -1 0.0 0.0 00 ?? BL 12:14PM 0:00.00 (hammer-S3) root -1 0.0 0.0 00 ?? BL 12:14PM 0:00.00 (hammer-S2) before giving the "hammer mirror-copy" command in /etc/rc.shutdown ?? also why are there 3 hammer mirror-stream processes? and what are the hammer-M, hammer-S* processes? Thankyou so much --Siju
Re: hammer mirror shows differrent results from differrent terminals
:Thanks a lot for your reply Matt :-) : :Could you please tell how mirror-stream actually works? :Does it work sequentially or as changes happen? : :By sequentially I mean If there are 100 files in the master then :mirror-stream copies from file 1 to 100 to the slave so if mirror has :passed from 1 to 50 and a change happens in file 25 it will be copied :to the slave only after the mirror-stream has completed copying from :50 till 100 and come back to file 25 from the begining. : :Again what happens if a file in the master and the corresponding file :in the slave are identical? Does mirror-stream copy it again or leave :it as it is? : :thanks : :Siju Mirror-stream works on HAMMER's B-Tree elements. Whenever a HAMMER flush occurs (every 30-60 seconds automatically or if you sync) any B-Tree elements which are deleted, updated, or inserted are sent across the wire to the slave. This works transactionally. Mirror-stream keeps track of the highest transaction id (TID) that it had successfully transfered and sends any B-Tree elements with a higher TID across the wire. When it gets an acknowledgement from the other side that they are on the slave's media it updates its notion of the highest transaction id and repeats. This is an incremental process. If a change is made to a file, such as new data appended (say a log file), only the changes are sent across the wire because only the B-Tree elements related to the new appends and the inode update will have been updated. So, for example, this will track a multi-gigabyte log file incrementally. Since mirror-stream only operates on records flushed to the media on the master one typically sees a little burst of activity every time the HAMMER filesystem is flushed (30-60 seconds, or sooner if there is a lot of activity). There is also a bandwidth-limiting option (-b) that you can set to put a ceiling on the network bandwidth the mirror-stream uses. And other parameters. Also, the mirroring code uses a non-queued implementation. There are no queue files involved which means the mirroring stream can be interrupted at any time and restarted at any time, even if it is days or months later, without losing track of its place. One can also mirror to multiple slaves or chain-mirror between slaves, and one can implement a different snapshot policy on the slaves then was on the master if one desires. You can use the verbose option (-v, -vv, -vvv, etc) to get more detail of the streaming operation or the quiet option (-q) to make it not output tracking info to stdout for use in a script. If you use a script to maintain the mirror-stream you typically either use cron to check if it is still running (using the lockf program to prevent duplication), or you create a while (1) loop in the script along with a long sleep (like sleep 300) to deal with any network interruptions which terminate the mirror-stream operation. -Matt Matthew Dillon
Re: hammer mirror shows differrent results from differrent terminals
On Wed, Jul 8, 2009 at 12:00 AM, Matthew Dillon wrote: > > That work is still in kernel memory. It hasn't been flushed to the > media yet. Mirroring operations only work on data flushed to the media. > This will occur automatically every 30-60 seconds or when you sync. > sync runs asynchronously though so it is best to use 'hammer synctid '. > Thanks a lot for your reply Matt :-) Could you please tell how mirror-stream actually works? Does it work sequentially or as changes happen? By sequentially I mean If there are 100 files in the master then mirror-stream copies from file 1 to 100 to the slave so if mirror has passed from 1 to 50 and a change happens in file 25 it will be copied to the slave only after the mirror-stream has completed copying from 50 till 100 and come back to file 25 from the begining. Again what happens if a file in the master and the corresponding file in the slave are identical? Does mirror-stream copy it again or leave it as it is? thanks Siju
Re: hammer mirror shows differrent results from differrent terminals
:Now in xterm1 I create a test file in the master pfs : :# touch test :# ls -l :total 0 :... :# hammer mirror-copy /Backup1/Data /Backup2/Data :Mirror-read: Mirror from 0001000204b1 to 0001000204f0 :Mirror-read /Backup1/Data succeeded :# :Now I check the slave in xterm2 : :# pwd :/Backup2/pfs/@@0x0001000204b0:1 :# ls :# :it shows nothing. :but if I check it with the full path ( in xterm2 ) it shows the file :is mirrored. :# ls -l /Backup2/Data/ :total 0 :-rw-r--r-- 1 root wheel 0 Jul 7 13:44 test : :Why is this so? Take a look at the transaction id's in the path. When you CD into the slave you are CD'ing into a snapshot on the slave. That is what that softlink is. It is dynamically generated by HAMMER but once you CD into it you are locked into a transaction id. So to see the changes you have to re-CD into the slave via the original softlink. All slave accesses are snapshot accesses, always. Accesses to master PFSs use a transaction id of -1 (0xULL) and are thus always 'current' accesses, requiring no re-cd. It is not a good idea to CD into a slave using a transaction id of -1 because you may catch ongoing mirroring operations as they are running, instead of after they've completed. :# rm test :# hammer mirror-copy /Backup1/Data /Backup2/Data :Mirror-read: Mirror from 000100020371 to 000100020370 :Mirror-read: No work to do :Mirror-read /Backup1/Data succeeded : :It says no work to do! :and the file is left in the slave. :... :thanks : :--Siju That work is still in kernel memory. It hasn't been flushed to the media yet. Mirroring operations only work on data flushed to the media. This will occur automatically every 30-60 seconds or when you sync. sync runs asynchronously though so it is best to use 'hammer synctid '. -Matt Matthew Dillon
