Re: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[Boris Liberman]

On 7/30/2010 12:13 PM, David Mann wrote:

For Windows I'd recommend taking a look at SyncToy.



Cheers,
Dave


Thanks, Dave, this tool proves optimal for some of my needs.

Boris

--
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

Re: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[steve harley]

On 2010-07-30 03:13 , David Mann wrote:

For Windows I'd recommend taking a look at SyncToy.


from what i see, SyncToy does not verify results (verification being the 
inspiration for this thread...)



--
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

Re: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[Mark Roberts]

David Mann wrote:

>For Windows I'd recommend taking a look at SyncToy.
>
>

Thanks for that recommendation, Dave. That's one useful little app!

-- 
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

Re: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[David Mann]

On Jul 30, 2010, at 9:11 AM, steve harley wrote:

> rsync is a protocol that has been implemented on several platforms; like a 
> lot else on your computer that started on Unix (world wide web ...), the 
> easiest route is to find a package that has already made it ready for 
> Windows; based on a hasty web search, DeltaCopy seems to be one of several 
> such packages:
> 
> 


For Windows I'd recommend taking a look at SyncToy.



Cheers,
Dave
-- 
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

Re: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[steve harley]

On 2010-07-29 11:55 , Sam L wrote:

Steve Harley wrote:
   rsync

 That seems to be a unix thing ... not sure I want to muck with
getting it to run on windows.



rsync is a protocol that has been implemented on several platforms; like 
a lot else on your computer that started on Unix (world wide web ...), 
the easiest route is to find a package that has already made it ready 
for Windows; based on a hasty web search, DeltaCopy seems to be one of 
several such packages:





--
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

Re: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[Doug Franklin]

On 2010-07-29 13:55, Sam L wrote:

 When I try to open the jpg using "windows picture and fax viewer"
I get a blank screen and
 either "generating preview" or "drawing failed".  When I use
"microsoft office picture viewer" I just get a black screen.
 And hey ... that's odd ... when I use "internet explorer" the file
actually seems to display.  Strange!


That potentially points toward something in the file formats that aren't 
being interpreted consistently among the different viewers, or something 
that's missing that some viewers can/will default and others won't, or 
they try but use a default value that doesn't work with these particular 
files.  JPG files are very sensitive to even small corruptions in the 
data stream due to the progressive nature of the encoding.


One way to get around it might be to open them in the program that can 
open them, and save them in that program, too.  That might "fill in the 
blanks" that are causing problems for the other programs.  Of course, 
you risk having another round of lossy-compression-degradation in the 
quality of the JPGs.



 Backup has been either a simple copy/paste using windows explorer
or by "syncback" which is a pretty decent
 free backup software.  File by file comparisons doesn't really
sound realistic or practical, except for maybe once a year.
 I have currently 15,000+ files.


15,000 files will take at most a few minutes to check on most modern 
computers using an external USB 2.0 or Firewire drive, less for an eSATA 
drive.  And even if it takes longer than that on your computer, kick off 
a backup Monday night when you go to bed, and a verification Tuesday 
night when you go to bed.  It'll be done in the morning.



 I just check my syncback software and there is a "verify" option
which I will use from not on at the very least.

   If it ain't verified, it ain't a backup.  Trusting that they were OK
may be part of what got you in this mess.


OK, I didn't make my meaning clear there.  Like someone else said 
earlier today, the only way to verify a backup is to restore it and 
actually use the restored files.


Now that I think about it, this actually might be a case where it would 
be a useful project, but still a small enough one that I could get it 
done, to create some Linux shell scripts and/or Windows "batch" files to 
do some of these operations.  Say, one to do the backup, one to do the 
restore, and one to test the restored files.


The easiest way for you to test your backups without writing scripts or 
programs would be to have another external drive that's doesn't hold 
anything else.  Fast format that drive, copy the backed up files to it, 
then use a free "duplicate file finder" utility to confirm the restored 
files are the same as the backed up files.


--
Thanks,
DougF (KG4LMZ)

--
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

Re: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[Sam L]

On Thu, Jul 29, 2010 at 2:34 PM, Bob W  wrote:
>>   I run xcopy as a scheduled task
>>
>>     Can you share a little more of what you do?  Did you write some bat or
> cmd
>> file and your schedule calls this file?
>
> that's right.
>
>>     Do you mind sharing your file with me (feel free to edit
>> files/paths/machine names etc if security is any concern)?
>>
>
> It's something along these lines. The script runs on a machine called
> PrecisionM50 and copies from 2 other machines, Vostro and Thinkpad, onto the
> local drive of PrecisionM50. When that's completed it copies what it's just
> done to an external drive (India) connected to PrecisionM50. The actual code
> that runs include error handling and a few other bits. The log file keeps a
> record so I know what happened.
>
> xcopy \\thinkpad\..\                      \\precisionm50\backups\thinkpad /d
> /s /c /i /f /g /r /k /x /y /z      >\\precisionm50\backuplog.txt
> xcopy \\vostro\..\                           \\precisionm50\backups\vostro
> /d /s /c /i /f /g /r /k /x /y /z     >>\\precisionm50\backuplog.txt
> xcopy \\precisionm50\backups \\precisionm50\india
> /d /s /c /i /f /g /r /k /x /y /z /j  >>\\precisionm50\backuplog.txt

Thanks ... Time to bone up on my xcopy switches!

On a positive note, the "windiff" pointer looks very promising.  It
was already on my laptop, and seems to do a good job of identifying
files in 2 directories that differ from each other.  And you can print
the results to a file.

That is pretty much exactly what I needed!

---
Sam

-- 
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

RE: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[Bob W]

>   I run xcopy as a scheduled task
> 
> Can you share a little more of what you do?  Did you write some bat or
cmd
> file and your schedule calls this file?

that's right.

> Do you mind sharing your file with me (feel free to edit
> files/paths/machine names etc if security is any concern)?
> 

It's something along these lines. The script runs on a machine called
PrecisionM50 and copies from 2 other machines, Vostro and Thinkpad, onto the
local drive of PrecisionM50. When that's completed it copies what it's just
done to an external drive (India) connected to PrecisionM50. The actual code
that runs include error handling and a few other bits. The log file keeps a
record so I know what happened.

xcopy \\thinkpad\..\  \\precisionm50\backups\thinkpad /d
/s /c /i /f /g /r /k /x /y /z  >\\precisionm50\backuplog.txt
xcopy \\vostro\..\   \\precisionm50\backups\vostro
/d /s /c /i /f /g /r /k /x /y /z >>\\precisionm50\backuplog.txt
xcopy \\precisionm50\backups \\precisionm50\india
/d /s /c /i /f /g /r /k /x /y /z /j  >>\\precisionm50\backuplog.txt



-- 
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

Re: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[Sam L]

> On Thu, Jul 29, 2010 at 7:14 AM, David Mann  wrote:
>> On Jul 29, 2010, at 9:02 AM, Doug Franklin wrote:
>
>> If it ain't verified, it ain't a backup.  Trusting that they were OK may be 
>> part of what got you in this mess.
>
> The only sure way to verify a backup is to restore from it.  On a test 
> system, of course.


Ok, let's see hopefully the questions and responses will be fairly
clear and google won't mangle my formatting.  :-\


Doug Franklin wrote:
  If it ain't off site, it ain't a backup.

I'm covered here.  Backups are at multiple sites.

  What does "will not open" mean?
  What messages do you get?
  Are these PEFs, JPGs, or what?
  What tool are you trying to use to open them?

When I try to open the jpg using "windows picture and fax viewer"
I get a blank screen and
either "generating preview" or "drawing failed".  When I use
"microsoft office picture viewer" I just get a black screen.
And hey ... that's odd ... when I use "internet explorer" the file
actually seems to display.  Strange!

  The scarier symptom is when the thumbnail appears to be fine but is in
  fact corrupt, because there is no easy way to verify if the files were
  backed up successfully.

  What do you mean?
  How are you backing up?
  Since you're just copying to an external drive, you can always do a
file-by-file comparison afterwards.

Sometimes even when the thumbnail displays an image in windows
explorer, if I click on the file to open it, I'll get
an error message (see above error messages).  So the scary part is
that even though the thumbnail seems to display
ok, the file is in fact corrupted.

Backup has been either a simple copy/paste using windows explorer
or by "syncback" which is a pretty decent
free backup software.  File by file comparisons doesn't really
sound realistic or practical, except for maybe once a year.
I have currently 15,000+ files.

  For the command-line inclined on Windows, the "comp" or "fc"
commands will do it, or simply add "/v" to the (x)copy
  command to start with. Similar options are available for Linux and
Mac, too.  For those preferring GUIs, there are a variety of
  free software tools out there for any given platform that will
compare the files after copying. WinDiff is one for Windows.

I just check my syncback software and there is a "verify" option
which I will use from not on at the very least.

  If it ain't verified, it ain't a backup.  Trusting that they were OK
may be part of what got you in this mess.

So true ...


Bob W wrote:
  I run xcopy as a scheduled task

Can you share a little more of what you do?  Did you write some
bat or cmd file and your schedule calls this file?
Do you mind sharing your file with me (feel free to edit
files/paths/machine names etc if security is any concern)?

Steve Harley wrote:
  rsync

That seems to be a unix thing ... not sure I want to muck with
getting it to run on windows.

PJ Alling wrote:
  You might want to check on the health of your hard drive

Good tip.  In this case the original problem was on a computer
that was retired long ago.
Current drives are all pretty new and reliable.


Boris Liberman wrote:
  EMC Retrospect

I looked online and it goes for $1670.  That is a scope creep!


---
Sam

-- 
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

Re: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[P. J. Alling]

What, you mean it's bad to restore from a bad backup?  Who'd a thought.

On 7/29/2010 7:14 AM, David Mann wrote:

On Jul 29, 2010, at 9:02 AM, Doug Franklin wrote:

   

If it ain't verified, it ain't a backup.  Trusting that they were OK may be 
part of what got you in this mess.
 

The only sure way to verify a backup is to restore from it.  On a test system, 
of course.

Cheers,
Dave
   



--
{\rtf1\ansi\ansicpg1252\deff0\deflang1033{\fonttbl{\f0\fnil\fcharset0 Courier 
New;}}
\viewkind4\uc1\pard\f0\fs20 I've just upgraded to Thunderbird 3.0 and the 
interface subtly weird.\par
}


--
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

Re: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[David Mann]

On Jul 29, 2010, at 9:02 AM, Doug Franklin wrote:

> If it ain't verified, it ain't a backup.  Trusting that they were OK may be 
> part of what got you in this mess.

The only sure way to verify a backup is to restore from it.  On a test system, 
of course.

Cheers,
Dave
-- 
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

Re: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[Boris Liberman]

On 7/29/2010 12:02 AM, Doug Franklin wrote:

On 2010-07-28 16:52, Sam L wrote:


My backup strategy so far has been very basic: [...]


If it ain't off site, it ain't a backup.


http://www.taobackup.com/

:-)

Boris

--
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

Re: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[Boris Liberman]

On 7/28/2010 11:52 PM, Sam L wrote:

Anyway:  is there any way that you can verify if a jpeg file is
working correctly / not corrupt?  Is there a way to verify that your
"copy" operation was successful?

(I have no evidence that the copy operation is what corrupted my
files, so maybe I should just continue on with my copy operations and
trust that they are okay).

Your thoughts on this are appreciated!


Sam, I am using EMC Retrospect software that came as a bundle with the 
network attached disk drive that I used to have. As integral part of 
backup operation it offers a number of options for verification. I 
normally choose the simplest way - after backup it would simply read 
both source and destination files and compare them for sameness.


Additionally, after my network drive expired electronically, I went on 
using the same program with simple USB HDD. Works like a charm.


HTH

Boris

--
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

Re: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[P. J. Alling]

You might want to check on the health of your hard drive.  A failing HD 
will lead to all kinds copy failures.


On 7/28/2010 5:30 PM, Bob W wrote:

My backup strategy so far has been very basic:  I have a couple windows xp
 

[...]
   

Anyway:  is there any way that you can verify if a jpeg file is working
 

correctly
   

/ not corrupt?  Is there a way to verify that your "copy" operation was
successful?

(I have no evidence that the copy operation is what corrupted my files, so
maybe I should just continue on with my copy operations and trust that
 

they
   

are okay).

Your thoughts on this are appreciated!
 

it depends on what you're using to copy the files. Any decent backup
software will give you the option of doing a read-after-write check, which
means the software reads each block back after it has written it, comparing
them to make sure the write was successful. It should also give a choice
about what to do if the raw check fails - e.g. retry, continue, ignore - and
send you a message about it, on the screen, in an email or maybe a series of
cryptic beeps.

You should check your copy operation thoroughly before you decide to just
continue with it.

I run xcopy as a scheduled task rather than backup or some other specialised
software because it's simple and puts an exact copy on the backup drive
rather than some humungous file in a non-obvious format, and is easy to
restore.

B



   



--
{\rtf1\ansi\ansicpg1252\deff0\deflang1033{\fonttbl{\f0\fnil\fcharset0 Courier 
New;}}
\viewkind4\uc1\pard\f0\fs20 I've just upgraded to Thunderbird 3.0 and the 
interface subtly weird.\par
}


--
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

Re: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[steve harley]

On 2010-07-28 14:52 , Sam L wrote:

This was mostly working fine but a while ago I realized a bunch of
files were corrupt, and I have no idea when or how they got corrupted.


one easy way to check integrity while backing up is with rsync, though 
it is not the ideal tool to check for corruption after a successful 
backup; i'm sure rsync can run on Windows, but i don't know the ins and outs


a long-term reliable backup system should probably include multiple 
backups and sets of checksums with which to periodically verify that 
each backup is intact; if one is found to have become corrupt over time, 
that media should probably be replaced with a new copy from an intact backup


(personally, i do use rsync and make multiple backups, but i don't keep 
checksums and check for corruption later)


--
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

RE: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[Bob W]

> My backup strategy so far has been very basic:  I have a couple windows xp
[...]
> 
> Anyway:  is there any way that you can verify if a jpeg file is working
correctly
> / not corrupt?  Is there a way to verify that your "copy" operation was
> successful?
> 
> (I have no evidence that the copy operation is what corrupted my files, so
> maybe I should just continue on with my copy operations and trust that
they
> are okay).
> 
> Your thoughts on this are appreciated!

it depends on what you're using to copy the files. Any decent backup
software will give you the option of doing a read-after-write check, which
means the software reads each block back after it has written it, comparing
them to make sure the write was successful. It should also give a choice
about what to do if the raw check fails - e.g. retry, continue, ignore - and
send you a message about it, on the screen, in an email or maybe a series of
cryptic beeps.

You should check your copy operation thoroughly before you decide to just
continue with it. 

I run xcopy as a scheduled task rather than backup or some other specialised
software because it's simple and puts an exact copy on the backup drive
rather than some humungous file in a non-obvious format, and is easy to
restore.

B



-- 
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

Re: Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[Doug Franklin]

On 2010-07-28 16:52, Sam L wrote:


My backup strategy so far has been very basic:  [...]


If it ain't off site, it ain't a backup.


1.  both thumbnail and full image will not open
2.  thumbnail will display, but full image will not open


What does "will not open" mean?  What messages do you get?  Are these 
PEFs, JPGs, or what?  What tool are you trying to use to open them?



The scarier symptom is when the thumbnail appears to be fine but is in
fact corrupt, because there is no easy way to verify if the files were
backed up successfully.


What do you mean?  How are you backing up?  Since you're just copying to 
an external drive, you can always do a file-by-file comparison afterwards.



Anyway:  is there any way that you can verify if a jpeg file is
working correctly / not corrupt?  Is there a way to verify that your
"copy" operation was successful?


For the command-line inclined on Windows, the "comp" or "fc" commands 
will do it, or simply add "/v" to the (x)copy command to start with. 
Similar options are available for Linux and Mac, too.  For those 
preferring GUIs, there are a variety of free software tools out there 
for any given platform that will compare the files after copying. 
WinDiff is one for Windows.



(I have no evidence that the copy operation is what corrupted my
files, so maybe I should just continue on with my copy operations and
trust that they are okay).


If it ain't verified, it ain't a backup.  Trusting that they were OK may 
be part of what got you in this mess.


--
Thanks,
DougF (KG4LMZ)

--
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.

Off Topic: how do you ensure your file backup operations are not corrupt? (windows xp)

[Sam L]

Hi all,

My backup strategy so far has been very basic:  I have a couple
windows xp computers and a couple of external drives.  I schedule a
backup of files from my official set of files to my external drive.
And I also manually copy the files from time to time to some other
drive.

This was mostly working fine but a while ago I realized a bunch of
files were corrupt, and I have no idea when or how they got corrupted.
 There are 2 symptoms of corruption:
1.  both thumbnail and full image will not open
2.  thumbnail will display, but full image will not open

The scarier symptom is when the thumbnail appears to be fine but is in
fact corrupt, because there is no easy way to verify if the files were
backed up successfully.

Anyway:  is there any way that you can verify if a jpeg file is
working correctly / not corrupt?  Is there a way to verify that your
"copy" operation was successful?

(I have no evidence that the copy operation is what corrupted my
files, so maybe I should just continue on with my copy operations and
trust that they are okay).

Your thoughts on this are appreciated!

---
Sam

-- 
PDML Pentax-Discuss Mail List
PDML@pdml.net
http://pdml.net/mailman/listinfo/pdml_pdml.net
to UNSUBSCRIBE from the PDML, please visit the link directly above and follow 
the directions.