Re: [ccp4bb] how to recover my data

[Ian Tickle]

Hi Zhijie

I checked the versions of todos & tounix that I used & it turns out that
the version of 'tounix' uses an even dumber method than the 'dumb' method
you describe.  It does the equivalent of:

perl -pe 's/\r//' original+todos.mtz > original+todos+tounix.mtz


This should normally be OK for a DOS text file since it should not contain
any \r except in the combination \r\n.  However for the MTZ file it would
be disastrous since there are likely to be many lone \r's in the binary
data.  In that case the code above removes the first \r on a line, instead
of the last one just before the \n.

I tried it with your method and I got perfect recovery of the original MTZ
file, as you observed.

Cheers

-- Ian

On 6 March 2015 at 17:28, Zhijie Li  wrote:

>   Hi Ian,
>
> Unix/Linux uses a single byte 0A (the linefeed character) as the line end
> marker for text, while DOS/Win use two bytes 0D 0A (carriage return + line
> feed). Old Mac systems use 0D. (what a mess!)
>
> So a simple UNIX to DOS operation should expand every 0A in a file to 0D
> 0A:
> perl -pe 's/\n/\r\n/' input.mtz > output.mtz
>
> For MTZ file, since the main body of it is an array of REAL*4, apparently
> this operation will generate a lot of ‘insertional mutations’. No wonder
> the data is scrambled and the header cannot be found (its position is
> shifted, no longer at where the pointer points).
>
> During the conversion from UNIX to DOS, there is a complication: there
> could be a few 0D 0A already in the original file. Smarter unix2dos
> programs will keep these 0D 0A unchanged so that there won’t be
> weird-looking 0D 0D 0A (for example
> http://www.thefreecountry.com/tofrodos/ states this in its source file
> and has this behavior). Some dumber methods will simply convert all of them
> into 0D 0D 0A, for example:
> perl -pe 's/\n/\r\n/' input.mtz > output.mtz
>
>
> When converting from the DOS format back to UNIX format, in our binary
> file case, the dumber method will work but the smarter programs will cause
> problems. Because for all the 0D 0A in the DOS file generated by the
> smarter programs, there is no way to tell which used to be 0A and which
> used to be 0D 0A in the original, and they are all converted back to 0A.
> Therefore some 0D in the original file will be lost. With the dumber
> method, all 0D 0A were 0A in the original file, so there would be no
> problem changing them all back to 0A.
>
>
> In a given MTZ, there will almost certainly be a lot of 0A. But 0D 0A
> could be rare or non-existent. So after a unix-dos then dos-unix
> conversion, the result depends on how many 0D 0A were there in the original
> file, and how the program did the UNIX-DOS conversion.
>
> With things like the following, it should be OK:
> perl -pe 's/\n/\r\n/' test.mtz > test1.mtz
> perl -pe 's/\r\n/\n/' test1.mtz > test2.mtz
>
> To test the dumber (perl) method, I used an MTZ file, which contains 7 0D
> 0A in the data section. Here is the result:
> test.mtz 2030184 bytes: MTZdump OK
> test1.mtz 2032486 bytes  : MTZdump error
> test2.mtz 2030184 bytes  : MTZdump OK
> cmp test.mtz test2.mtz : the two files are identical
>
> With todos/fromdos (http://www.thefreecountry.com/tofrodos/ ):
> test.mtz 2030184 bytes : MTZdump OK
> todos.mtz 2032479 bytes : MTZdump error (note the size difference compared
> to test1.mtz, the 7 0D 0A in the original file were kept unchanged)
> fromdos.mtz 2030177 bytes : MTZdump error (7 original 0D 0A were shrinked
> to 0A)
>
>
> Ian, in your test with todos and tounix, it seems that the final MTZ still
> has a header information at the correct location, so that MTZdump could
> read it. But some of the numbers saved in the data array seem damaged, so
> some of the stats in MTZdump were out of range. It would be interesting to
> read the todos and tounix source code to see why that happened. Or with a
> binary file comparison tool we might be able to guess the cause by taking a
> look at the two files.
>
> Zhijie
>
>
>
>
>
>
>  *From:* Ian Tickle 
> *Sent:* Friday, March 06, 2015 5:59 AM
> *To:* CCP4BB@JISCMAIL.AC.UK
> *Subject:* Re: [ccp4bb] how to recover my data
>
> Hi, just for fun and to demonstrate what can go horribly wrong if you
> blindly use utilities that were specifically designed for changing the line
> terminators in an ASCII file, I applied these utilities to an MTZ file.
>
> First I used 'todos' to simulate what I suspect the technician has done:
>
> todos < original.mtz > original+todos.mtz
>
> Then I used 'tounix' in an attempt to recover the original file, as others
> have suggested:
>
> tounix < 

Re: [ccp4bb] how to recover my data

[Smith Liu]

Dear All,

 
For the corrupted mtz file, if we open it by notepad, it would be messy code or 
unreadble code.
 
In addition, not all the mtz files in the damaged hardware were corrupted, some 
mts files are still readable by Coot.
 
Do you have any suggestions to correct the unreadbale mtz files to readable?
 
Smith








At 2015-03-07 22:51:12, "Zhijie Li"  wrote:

Smith,
 
As Ian said, mtzdump can give you some clues, but will not solve your problem.
 
The file that mtzdump could not read apparently has changed length due to some 
sort of insertion. The other file that mtzdump could process has the correct 
length and the stats look reasonable, so it might be OK, although we still can 
not rule out the possibility that a few numbers might have been changed in the 
dataset. Have you tried to do anything with the latter file, such as a few 
rounds of refinement? What did you see?
 
If you send me the two mtz I can take a look and make some guesses. If the 
problem was as simple as introduction of some 0D 0A insertion/deletion it can 
be fixed. If you can consult the technician who recovered the data and tell us 
what software he used it would be helpful.
 
Zhijie
 
 
 
From:Smith Liu
Sent: Saturday, March 07, 2015 9:24 AM
To:CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] how to recover my data
 
Thanks Ian. I got it from the bin folder you mentioned. A moment ago I have 
failed to localize it from the CCP4 graphical interface.
 
Smith





 
At 2015-03-07 21:13:58, "Ian Tickle"  wrote:

Hi Smith


Not sure what you mean, the current version of CCP4 (as have all previous 
versions) certainly does contain mtzdump:


> which mtzdump
/software/CCP4-6.5.0/ccp4-6.5/bin/mtzdump


The "limited output" is intentional: mtzdump by default only lists the first 10 
reflexions, and you have exactly 10 in your list.  If you want more then supply 
an NREF value, e.g.


echo nref 9 |mtzdump HKLIN my.mtz |less


But mtzdump (or any MTZ utility for that matter) is not going to help you with 
a corrupted file, since the MTZ read routines assume that the file has a 
strictly defined format which has most likely been modified when it was 
transferred from the damaged disk, and is therefore now unreadable by any 
program that assumes the defined format.


What does the technician who transferred the data have to say about it?  He is 
the only person who can possibly tell you how the file has been corrupted.  
What you need is a hex dump of the file (e.g. using 'hexdump') and someone who 
understands a) the correct format and b) exactly how it was corrupted.  Then it 
may be possible to write a bespoke program tor recover your data.  But that is 
not going to be easy: it will require someone with some expertise in 
programming.

Attempting to read the file with random programs has even less chance of 
working than I have of winning the lottery!  As others have suggested there are 
much easier ways of recovering your data (such as reprocessing the images).


Cheers


-- Ian

 
 
On 7 March 2015 at 06:29, Smith Liu  wrote:

Dear All,
 
The current version of CCP4 does not contain mtzdump. I use UCLA MBI — mtzdump 
to process it. For a mtz file, the output says there was no valid reflection 
data. For another mtz file, it only gave about 50 lines information (not a new 
mtz file). As for the original mtz is very large, I think at least some message 
in the mtz file has been missing or has not been recovered by UCLA MBI — 
mtzdump.
 
Please feel free for further advise.
 
Smith
 
 
Follwing: output from the UCLAMBI-mtzdump



mtzdump - dump data from an MTZ reflection data file. A CCP4 program
Your mtzdump Results
Cell Dimensions : (obsolete - refer to dataset cell dimensions above)

  130.3260  130.3260  360.3400   90.   90.   90. 

 *  Resolution Range :

0.30.22676 (180.170 -  2.100 A )

 * Sort Order :

  1 2 3 0 0

 * Space group = 'I 4 2 2' (number 97)



 
| |
|
1 ASC  0  62  0  100.00 32.5 32.5 180.17   2.10   H  H
| |
|
2 NONE 0  43  0  100.00 13.4 13.4 180.17   2.10   H  K
| |
|
3 NONE 0 171  0  100.00 64.4 64.4 180.17   2.10   H  L
| |
|
4 NONE0.019.0 0  100.00 9.51 9.51 180.17   2.10   I  
FreeR_flag
| |
|
5 NONE   65.8466807.5 47890   47.08 14985.84 14985.84  41.21   2.10   J  IMEAN
| |
|
6 NONE   22.7 37273.6 47890   47.08  1161.73  1161.73  41.21   2.10   Q  
SIGIMEAN
| |
|
7 NONE   65.8466807.5 50197   44.54 15601.05 15601.05  41.21   2.10   K  I(+)
| |
|
8 NONE   36.9 37273.6 50197   44.54  1723.38  1723.38  41.21   2.10   M  SIGI(+)
| |
|
9 NONE   65.8466807.5 53778   40.58 16626.28 16626.28  41.21   2.10   K  I(-)
| |
|
10 NONE   36.9 37273.6 53778   40.58  1568.40  1568.40  41.21   2.10   M  
SIGI(-)
| |
|
11 NONE   67.3  6770.8 47890   47.08  1006.23  1006.23  41.21   2.10   F  F
| |
|
12 NONE5.6   446.6 47890   47.

Re: [ccp4bb] how to recover my data

[Zhijie Li]

Smith,

As Ian said, mtzdump can give you some clues, but will not solve your problem. 

The file that mtzdump could not read apparently has changed length due to some 
sort of insertion. The other file that mtzdump could process has the correct 
length and the stats look reasonable, so it might be OK, although we still can 
not rule out the possibility that a few numbers might have been changed in the 
dataset. Have you tried to do anything with the latter file, such as a few 
rounds of refinement? What did you see?

If you send me the two mtz I can take a look and make some guesses. If the 
problem was as simple as introduction of some 0D 0A insertion/deletion it can 
be fixed. If you can consult the technician who recovered the data and tell us 
what software he used it would be helpful.

Zhijie



From: Smith Liu 
Sent: Saturday, March 07, 2015 9:24 AM
To: CCP4BB@JISCMAIL.AC.UK 
Subject: Re: [ccp4bb] how to recover my data

Thanks Ian. I got it from the bin folder you mentioned. A moment ago I have 
failed to localize it from the CCP4 graphical interface.

Smith






At 2015-03-07 21:13:58, "Ian Tickle"  wrote:

  Hi Smith


  Not sure what you mean, the current version of CCP4 (as have all previous 
versions) certainly does contain mtzdump:


  > which mtzdump
  /software/CCP4-6.5.0/ccp4-6.5/bin/mtzdump


  The "limited output" is intentional: mtzdump by default only lists the first 
10 reflexions, and you have exactly 10 in your list.  If you want more then 
supply an NREF value, e.g.


  echo nref 9 |mtzdump HKLIN my.mtz |less


  But mtzdump (or any MTZ utility for that matter) is not going to help you 
with a corrupted file, since the MTZ read routines assume that the file has a 
strictly defined format which has most likely been modified when it was 
transferred from the damaged disk, and is therefore now unreadable by any 
program that assumes the defined format.


  What does the technician who transferred the data have to say about it?  He 
is the only person who can possibly tell you how the file has been corrupted.  
What you need is a hex dump of the file (e.g. using 'hexdump') and someone who 
understands a) the correct format and b) exactly how it was corrupted.  Then it 
may be possible to write a bespoke program tor recover your data.  But that is 
not going to be easy: it will require someone with some expertise in 
programming.

  Attempting to read the file with random programs has even less chance of 
working than I have of winning the lottery!  As others have suggested there are 
much easier ways of recovering your data (such as reprocessing the images).


  Cheers


  -- Ian



  On 7 March 2015 at 06:29, Smith Liu  wrote:

Dear All,

The current version of CCP4 does not contain mtzdump. I use UCLA MBI ― 
mtzdump to process it. For a mtz file, the output says there was no valid 
reflection data. For another mtz file, it only gave about 50 lines information 
(not a new mtz file). As for the original mtz is very large, I think at least 
some message in the mtz file has been missing or has not been recovered by UCLA 
MBI ― mtzdump.

Please feel free for further advise.

Smith


Follwing: output from the UCLAMBI-mtzdump



mtzdump - dump data from an MTZ reflection data file. A CCP4 program
Your mtzdump Results
Cell Dimensions : (obsolete - refer to dataset cell dimensions above)

  130.3260  130.3260  360.3400   90.   90.   90. 

 *  Resolution Range :

0.30.22676 (180.170 -  2.100 A )

 * Sort Order :

  1 2 3 0 0

 * Space group = 'I 4 2 2' (number 97)



  
1 ASC  0  62  0  100.00 32.5 32.5 180.17   2.10   H  H  
2 NONE 0  43  0  100.00 13.4 13.4 180.17   2.10   H  K  
3 NONE 0 171  0  100.00 64.4 64.4 180.17   2.10   H  L  
4 NONE0.019.0 0  100.00 9.51 9.51 180.17   2.10   I  
FreeR_flag  
5 NONE   65.8466807.5 47890   47.08 14985.84 14985.84  41.21   2.10   J  IMEAN  
6 NONE   22.7 37273.6 47890   47.08  1161.73  1161.73  41.21   2.10   Q  
SIGIMEAN  
7 NONE   65.8466807.5 50197   44.54 15601.05 15601.05  41.21   2.10   K  I(+)  
8 NONE   36.9 37273.6 50197   44.54  1723.38  1723.38  41.21   2.10   M  
SIGI(+)  
9 NONE   65.8466807.5 53778   40.58 16626.28 16626.28  41.21   2.10   K  I(-)  
10 NONE   36.9 37273.6 53778   40.58  1568.40  1568.40  41.21   2.10   M  
SIGI(-)  
11 NONE   67.3  6770.8 47890   47.08  1006.23  1006.23  41.21   2.10   F  F  
12 NONE5.6   446.6 47890   47.0876.6976.69  41.21   2.10   Q  SIGF  
13 NONE -514.8   537.4 56085   38.03 0.2187.57  41.21   2.10   D  DANO  
14 NONE0.0   380.1 56085   38.0391.6391.63  41.21   2.10   Q  
SIGDANO  
15 NONE   67.3  6770.8 50197   44.54  1032.56  1032.56  41.21   2.10   G  F(+)  
16 NONE8.2   446.6 50197   44.5490.1390.13  41.21   2.10   L  
SIGF(+)  
17

Re: [ccp4bb] how to recover my data

[Smith Liu]

? ?  
0   0   8   18.00   ? ? ? ? ?  
  ? ? ? ? ? ?  
  ? ? ? ?  
0   0  103.00   ? ? ? ? ?  
  ? ? ? ? ? ?  
  ? ? ? ?  
0   0  12   17.00   3090.57268.91   3090.57268.91   3090.57
   268.91554.30 24.36  0.00  0.00554.30
24.36554.30 24.36  1.00
0   0  149.00  23242.80   1362.28  23242.80   1362.28  23242.80
  1362.28   1522.33 44.83  0.00  0.00   1522.33
44.83   1522.33 44.83  1.00
0   0  165.00  29139.17   1697.95  29139.17   1697.95  29139.17
  1697.95   1704.50 49.90  0.00  0.00   1704.50
49.90   1704.50 49.90  1.00
0   0  18   12.00478.99 57.42478.99 57.42478.99
57.42217.65 13.29  0.00  0.00217.65
13.29217.65 13.29  1.00
0   0  203.00 113229.90   6443.99 113229.90   6443.99 113229.90
  6443.99   3358.69 96.10  0.00  0.00   3358.69
96.10   3358.69 96.10  1.00
 end, thus very limited information output



At 2015-03-06 15:56:19, "Tim Gruene"  wrote:
>-BEGIN PGP SIGNED MESSAGE-
>Hash: SHA1
>
>Dear Smith,
>
>an mtz-file is a binary file and none of the programs you list is
>suitable to open it in a meaningful manner.
>
>Why don't you follow the advice you were already given?
>mtzdump from ccp4 would be the first program to try and see if any
>information can be retrieved from the mtz-file.
>
>Regards,
>Tim
>
>On 03/06/2015 08:20 AM, Smith Lee wrote:
>> Dear All, For the issue of the recovery of the mtz file, I have
>> tried randomly to use excel to open one specific mtz file, however
>> in this way all  the mtz files in the computer will have a excel
>> icon (X), although the file extension is still .mtz. If I tried
>> further to open one specific mtz file (with excel icon) with the
>> notepad, all the mtz files will have the notepad icon. If I tried
>> further to open one specific mtz file (with notepad icon) with the
>> wordpad, all the mtz files will have the wordpad icon.  I hope
>> these cluses can be helpful for you to give me the advise on
>> recovery of mtz files. Smith
>> 
>> On Thursday, March 5, 2015 11:51 PM, Robbie Joosten
>>  wrote:
>> 
>> 
>> Hi Smith,
>> 
>> If this is really the problem Ian describes, you can try the Linux
>> programs unix2dos and dos2unix the change the line endings. A
>> potential source of the problem might be copying the file with
>> certain (S)FTP clients: in 'text-mode' they change the line endings
>> to your OS default to be user friendly.
>> 
>> Cheers, Robbie
>> 
>>> -Original Message- From: CCP4 bulletin board
>>> [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian Tickle Sent:
>>> Thursday, March 05, 2015 14:04 To: CCP4BB@JISCMAIL.AC.UK Subject:
>>> Re: [ccp4bb] how to recover my data
>>> 
>>> Hi Smith
>>> 
>>> 
>>> I sympathise with your plight - I have had to do similar things
>>> in the past for other people!  I think your most fruitful course
>>> of action would be to talk to the technician who recovered your
>>> data because only he knows what he actually did to recover it.
>>> 
>>> 
>>> From your description of your recovery of the PDB file it looks
>>> to me like a line terminator issue, i.e. was the original file
>>> created in Linux, Windows or Mac?  This is relevant because the
>>> line terminators are different and it sounds like the technician
>>> didn't simply copy the file, he changed the line terminators.  If
>>> he did the same with the MTZ file thinking it was a text file the
>>> additional line terminators would corrupt the binary data making
>>> it impossible to read with any of the CCP4 MTZ utilities.  If you
>>> can understand exactly what the technician did you may be able to
>>> reverse it and recover the binary data.
>>> 
>>> 
>>> Hope this helps!
>>> 
>>> 
>>> Cheers
>>> 
>>> 
>>> -- Ian
>>> 
>>> 
>>> On 5 March 2015 at 05:36, Smith Lee <0459ef8548d5-dmarc- 
>>> requ...@jiscmail

Re: [ccp4bb] how to recover my data

[Ian Tickle]

? ? ? ? ? ?
>   ? ? ? ?
> 0   0   69.00   ? ? ? ? ?
>   ? ? ? ? ? ?
>   ? ? ? ?
> 0   0   8   18.00   ? ? ? ? ?
>   ? ? ? ? ? ?
>   ? ? ? ?
> 0   0  103.00   ? ? ? ? ?
>   ? ? ? ? ? ?
>   ? ? ? ?
> 0   0  12   17.00   3090.57268.91   3090.57268.91   3090.57
>268.91554.30 24.36  0.00  0.00554.30
> 24.36554.30 24.36  1.00
> 0   0  149.00  23242.80   1362.28  23242.80   1362.28  23242.80
>   1362.28   1522.33 44.83  0.00  0.00   1522.33
> 44.83   1522.33 44.83  1.00
> 0   0  165.00  29139.17   1697.95  29139.17   1697.95  29139.17
>   1697.95   1704.50 49.90  0.00  0.00   1704.50
> 49.90   1704.50 49.90  1.00
> 0   0  18   12.00478.99 57.42478.99 57.42478.99
> 57.42217.65 13.29  0.00  0.00217.65
> 13.29217.65 13.29  1.00
> 0   0  203.00 113229.90   6443.99 113229.90   6443.99 113229.90
>   6443.99   3358.69 96.10  0.00  0.00   3358.69
> 96.10   3358.69 96.10  1.00
>  end, thus very limited information output
>
>
>
> At 2015-03-06 15:56:19, "Tim Gruene"  wrote:
> >-BEGIN PGP SIGNED MESSAGE-
> >Hash: SHA1
> >
> >Dear Smith,
> >
> >an mtz-file is a binary file and none of the programs you list is
> >suitable to open it in a meaningful manner.
> >
> >Why don't you follow the advice you were already given?
> >mtzdump from ccp4 would be the first program to try and see if any
> >information can be retrieved from the mtz-file.
> >
> >Regards,
> >Tim
> >
> >On 03/06/2015 08:20 AM, Smith Lee wrote:
> >> Dear All, For the issue of the recovery of the mtz file, I have
> >> tried randomly to use excel to open one specific mtz file, however
> >> in this way all  the mtz files in the computer will have a excel
> >> icon (X), although the file extension is still .mtz. If I tried
> >> further to open one specific mtz file (with excel icon) with the
> >> notepad, all the mtz files will have the notepad icon. If I tried
> >> further to open one specific mtz file (with notepad icon) with the
> >> wordpad, all the mtz files will have the wordpad icon.  I hope
> >> these cluses can be helpful for you to give me the advise on
> >> recovery of mtz files. Smith
> >>
> >> On Thursday, March 5, 2015 11:51 PM, Robbie Joosten
> >>  wrote:
> >>
> >>
> >> Hi Smith,
> >>
> >> If this is really the problem Ian describes, you can try the Linux
> >> programs unix2dos and dos2unix the change the line endings. A
> >> potential source of the problem might be copying the file with
> >> certain (S)FTP clients: in 'text-mode' they change the line endings
> >> to your OS default to be user friendly.
> >>
> >> Cheers, Robbie
> >>
> >>> -Original Message- From: CCP4 bulletin board
> >>> [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian Tickle Sent:
> >>> Thursday, March 05, 2015 14:04 To: CCP4BB@JISCMAIL.AC.UK Subject:
> >>> Re: [ccp4bb] how to recover my data
> >>>
> >>> Hi Smith
> >>>
> >>>
> >>> I sympathise with your plight - I have had to do similar things
> >>> in the past for other people!  I think your most fruitful course
> >>> of action would be to talk to the technician who recovered your
> >>> data because only he knows what he actually did to recover it.
> >>>
> >>>
> >>> From your description of your recovery of the PDB file it looks
> >>> to me like a line terminator issue, i.e. was the original file
> >>> created in Linux, Windows or Mac?  This is relevant because the
> >>> line terminators are different and it sounds like the technician
> >>> didn't simply c

Re: [ccp4bb] how to recover my data

[Jon Agirre]

>
> The current version of CCP4 does not contain mtzdump.
>

It surely does on my machine:

$ which mtzdump
/Applications/ccp4-6.5/bin/mtzdump

-- 
Dr Jon Agirre
York Structural Biology Laboratory / Department of Chemistry
University of York, Heslington, YO10 5DD, York, England
http://www.york.ac.uk/chemistry/research/ysbl/people/research/jagirre/
+44 (0) 1904 32 8253

Re: [ccp4bb] how to recover my data

[Smith Liu]

t;
>Why don't you follow the advice you were already given?
>mtzdump from ccp4 would be the first program to try and see if any
>information can be retrieved from the mtz-file.
>
>Regards,
>Tim
>
>On 03/06/2015 08:20 AM, Smith Lee wrote:
>> Dear All, For the issue of the recovery of the mtz file, I have
>> tried randomly to use excel to open one specific mtz file, however
>> in this way all  the mtz files in the computer will have a excel
>> icon (X), although the file extension is still .mtz. If I tried
>> further to open one specific mtz file (with excel icon) with the
>> notepad, all the mtz files will have the notepad icon. If I tried
>> further to open one specific mtz file (with notepad icon) with the
>> wordpad, all the mtz files will have the wordpad icon.  I hope
>> these cluses can be helpful for you to give me the advise on
>> recovery of mtz files. Smith
>> 
>> On Thursday, March 5, 2015 11:51 PM, Robbie Joosten
>>  wrote:
>> 
>> 
>> Hi Smith,
>> 
>> If this is really the problem Ian describes, you can try the Linux
>> programs unix2dos and dos2unix the change the line endings. A
>> potential source of the problem might be copying the file with
>> certain (S)FTP clients: in 'text-mode' they change the line endings
>> to your OS default to be user friendly.
>> 
>> Cheers, Robbie
>> 
>>> -Original Message- From: CCP4 bulletin board
>>> [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian Tickle Sent:
>>> Thursday, March 05, 2015 14:04 To: CCP4BB@JISCMAIL.AC.UK Subject:
>>> Re: [ccp4bb] how to recover my data
>>> 
>>> Hi Smith
>>> 
>>> 
>>> I sympathise with your plight - I have had to do similar things
>>> in the past for other people!  I think your most fruitful course
>>> of action would be to talk to the technician who recovered your
>>> data because only he knows what he actually did to recover it.
>>> 
>>> 
>>> From your description of your recovery of the PDB file it looks
>>> to me like a line terminator issue, i.e. was the original file
>>> created in Linux, Windows or Mac?  This is relevant because the
>>> line terminators are different and it sounds like the technician
>>> didn't simply copy the file, he changed the line terminators.  If
>>> he did the same with the MTZ file thinking it was a text file the
>>> additional line terminators would corrupt the binary data making
>>> it impossible to read with any of the CCP4 MTZ utilities.  If you
>>> can understand exactly what the technician did you may be able to
>>> reverse it and recover the binary data.
>>> 
>>> 
>>> Hope this helps!
>>> 
>>> 
>>> Cheers
>>> 
>>> 
>>> -- Ian
>>> 
>>> 
>>> On 5 March 2015 at 05:36, Smith Lee <0459ef8548d5-dmarc- 
>>> requ...@jiscmail.ac.uk> wrote:
>>> 
>>> 
>>> 
>>> Dear All,
>>> 
>>> Recently my computer hardware has been broken and all the data 
>>> has been recovered to movable hardware by technician. However I
>>> find the recovered PDB file and the MTZcould not be openned by
>>> Coot. Then I open the revovered PDB file by WordPad, and from
>>> WordPad I copied it to notepad and save it as pdb file. I find
>>> the Coot can open the notepad saved pdb file, thus my pdb files
>>> can be succesfully recovered from the hardware.
>>> 
>>> But will you please tell me how to have Coot open my mtz file?
>>> After data recovery by the technicial, the data size of the mtz
>>> file did not decrease, thus I think there is a way to have it
>>> recovered.
>>> 
>>> I have not noticed there were similar or identical posts as mine
>>> for recovery data before in the CCP4 mail list.
>>> 
>>> Thus I am looking forward to getting a reply from you on how to 
>>> recover my mtz file.
>>> 
>>> 
>>> Smith
>>> 
>>> 
>> 
>> 
>> 
>
>- -- 
>- --
>Dr Tim Gruene
>Institut fuer anorganische Chemie
>Tammannstr. 4
>D-37077 Goettingen
>
>GPG Key ID = A46BEE1A
>
>-BEGIN PGP SIGNATURE-
>Version: GnuPG v1
>
>iD8DBQFU+V2jUxlJ7aRr7hoRApjyAKCwUB2lXaXFEoFievrLgJ+YDPZg+gCcDF6x
>IkBOMnu3OH/gjJknjvPWP+I=
>=K7G9
>-END PGP SIGNATURE-

Re: [ccp4bb] how to recover my data

[Zhijie Li]

Hi Ian,

Unix/Linux uses a single byte 0A (the linefeed character) as the line end 
marker for text, while DOS/Win use two bytes 0D 0A (carriage return + line 
feed). Old Mac systems use 0D. (what a mess!) 

So a simple UNIX to DOS operation should expand every 0A in a file to 0D 0A:
perl -pe 's/\n/\r\n/' input.mtz > output.mtz 

For MTZ file, since the main body of it is an array of REAL*4, apparently this 
operation will generate a lot of ‘insertional mutations’. No wonder the data is 
scrambled and the header cannot be found (its position is shifted, no longer at 
where the pointer points).

During the conversion from UNIX to DOS, there is a complication: there could be 
a few 0D 0A already in the original file. Smarter unix2dos programs will keep 
these 0D 0A unchanged so that there won’t be weird-looking 0D 0D 0A (for 
example http://www.thefreecountry.com/tofrodos/ states this in its source file 
and has this behavior). Some dumber methods will simply convert all of them 
into 0D 0D 0A, for example: 
perl -pe 's/\n/\r\n/' input.mtz > output.mtz


When converting from the DOS format back to UNIX format, in our binary file 
case, the dumber method will work but the smarter programs will cause problems. 
Because for all the 0D 0A in the DOS file generated by the smarter programs, 
there is no way to tell which used to be 0A and which used to be 0D 0A in the 
original, and they are all converted back to 0A. Therefore some 0D in the 
original file will be lost. With the dumber method, all 0D 0A were 0A in the 
original file, so there would be no problem changing them all back to 0A.


In a given MTZ, there will almost certainly be a lot of 0A. But 0D 0A could be 
rare or non-existent. So after a unix-dos then dos-unix conversion, the result 
depends on how many 0D 0A were there in the original file, and how the program 
did the UNIX-DOS conversion.

With things like the following, it should be OK:
perl -pe 's/\n/\r\n/' test.mtz > test1.mtz
perl -pe 's/\r\n/\n/' test1.mtz > test2.mtz

To test the dumber (perl) method, I used an MTZ file, which contains 7 0D 0A in 
the data section. Here is the result:
test.mtz 2030184 bytes: MTZdump OK
test1.mtz 2032486 bytes  : MTZdump error
test2.mtz 2030184 bytes  : MTZdump OK
cmp test.mtz test2.mtz : the two files are identical

With todos/fromdos (http://www.thefreecountry.com/tofrodos/ ):
test.mtz 2030184 bytes : MTZdump OK
todos.mtz 2032479 bytes : MTZdump error (note the size difference compared to 
test1.mtz, the 7 0D 0A in the original file were kept unchanged)
fromdos.mtz 2030177 bytes : MTZdump error (7 original 0D 0A were shrinked to 0A)


Ian, in your test with todos and tounix, it seems that the final MTZ still has 
a header information at the correct location, so that MTZdump could read it. 
But some of the numbers saved in the data array seem damaged, so some of the 
stats in MTZdump were out of range. It would be interesting to read the todos 
and tounix source code to see why that happened. Or with a binary file 
comparison tool we might be able to guess the cause by taking a look at the two 
files.

Zhijie






From: Ian Tickle 
Sent: Friday, March 06, 2015 5:59 AM
To: CCP4BB@JISCMAIL.AC.UK 
Subject: Re: [ccp4bb] how to recover my data

Hi, just for fun and to demonstrate what can go horribly wrong if you blindly 
use utilities that were specifically designed for changing the line terminators 
in an ASCII file, I applied these utilities to an MTZ file.


First I used 'todos' to simulate what I suspect the technician has done:


todos < original.mtz > original+todos.mtz


Then I used 'tounix' in an attempt to recover the original file, as others have 
suggested:

tounix < original+todos.mtz > original+todos+tounix.mtz


What can possibly go wrong?  The log files from mtzdump are attached - see for 
yourself! (the mtzdump on original+todos.mtz went into an infinite loop and I 
had to kill the process).


Cheers


-- Ian





On 6 March 2015 at 07:20, Smith Lee 
<0459ef8548d5-dmarc-requ...@jiscmail.ac.uk> wrote:

  Dear All,

  For the issue of the recovery of the mtz file, I have tried randomly to use 
excel to open one specific mtz file, however in this way all  the mtz files in 
the computer will have a excel icon (X), although the file extension is still 
.mtz. If I tried further to open one specific mtz file (with excel icon) with 
the notepad, all the mtz files will have the notepad icon. If I tried further 
to open one specific mtz file (with notepad icon) with the wordpad, all the mtz 
files will have the wordpad icon. 

  I hope these cluses can be helpful for you to give me the advise on recovery 
of mtz files.

  Smith



  On Thursday, March 5, 2015 11:51 PM, Robbie Joosten 
 wrote:




  Hi Smith,

  If this is really the problem Ian describes, you can try the Linux programs 
unix2dos and dos2unix the change the line endings. A potent

Re: [ccp4bb] how to recover my data

[Ian Tickle]

Hi, just for fun and to demonstrate what can go horribly wrong if you
blindly use utilities that were specifically designed for changing the line
terminators in an ASCII file, I applied these utilities to an MTZ file.

First I used 'todos' to simulate what I suspect the technician has done:

todos < original.mtz > original+todos.mtz

Then I used 'tounix' in an attempt to recover the original file, as others
have suggested:

tounix < original+todos.mtz > original+todos+tounix.mtz

What can possibly go wrong?  The log files from mtzdump are attached - see
for yourself! (the mtzdump on original+todos.mtz went into an infinite loop
and I had to kill the process).

Cheers

-- Ian



On 6 March 2015 at 07:20, Smith Lee <
0459ef8548d5-dmarc-requ...@jiscmail.ac.uk> wrote:

> Dear All,
>
> For the issue of the recovery of the mtz file, I have tried randomly to
> use excel to open one specific mtz file, however in this way all  the mtz
> files in the computer will have a excel icon (X), although the file
> extension is still .mtz. If I tried further to open one specific mtz file
> (with excel icon) with the notepad, all the mtz files will have the notepad
> icon. If I tried further to open one specific mtz file (with notepad icon)
> with the wordpad, all the mtz files will have the wordpad icon.
>
> I hope these cluses can be helpful for you to give me the advise on
> recovery of mtz files.
>
> Smith
>
>
>   On Thursday, March 5, 2015 11:51 PM, Robbie Joosten <
> robbie_joos...@hotmail.com> wrote:
>
>
> Hi Smith,
>
> If this is really the problem Ian describes, you can try the Linux
> programs unix2dos and dos2unix the change the line endings. A potential
> source of the problem might be copying the file with certain (S)FTP
> clients: in 'text-mode' they change the line endings to your OS default to
> be user friendly.
>
> Cheers,
> Robbie
>
> > -Original Message-
> > From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of
> > Ian Tickle
> > Sent: Thursday, March 05, 2015 14:04
> > To: CCP4BB@JISCMAIL.AC.UK
> > Subject: Re: [ccp4bb] how to recover my data
> >
> > Hi Smith
> >
> >
> > I sympathise with your plight - I have had to do similar things in the
> past for
> > other people!  I think your most fruitful course of action would be to
> talk to
> > the technician who recovered your data because only he knows what he
> > actually did to recover it.
> >
> >
> > From your description of your recovery of the PDB file it looks to me
> like a
> > line terminator issue, i.e. was the original file created in Linux,
> Windows or
> > Mac?  This is relevant because the line terminators are different and it
> > sounds like the technician didn't simply copy the file, he changed the
> line
> > terminators.  If he did the same with the MTZ file thinking it was a
> text file
> > the additional line terminators would corrupt the binary data making it
> > impossible to read with any of the CCP4 MTZ utilities.  If you can
> understand
> > exactly what the technician did you may be able to reverse it and
> recover the
> > binary data.
> >
> >
> > Hope this helps!
> >
> >
> > Cheers
> >
> >
> > -- Ian
> >
> >
> > On 5 March 2015 at 05:36, Smith Lee <0459ef8548d5-dmarc-
> > requ...@jiscmail.ac.uk> wrote:
> >
> >
> >
> > Dear All,
> >
> > Recently my computer hardware has been broken and all the data
> > has been recovered to movable hardware by technician. However I find the
> > recovered PDB file and the MTZcould not be openned by Coot. Then I open
> > the revovered PDB file by WordPad, and from WordPad I copied it to
> > notepad and save it as pdb file. I find the Coot can open the notepad
> saved
> > pdb file, thus my pdb files can be succesfully recovered from the
> hardware.
> >
> > But will you please tell me how to have Coot open my mtz file? After
> > data recovery by the technicial, the data size of the mtz file did not
> decrease,
> > thus I think there is a way to have it recovered.
> >
> > I have not noticed there were similar or identical posts as mine for
> > recovery data before in the CCP4 mail list.
> >
> > Thus I am looking forward to getting a reply from you on how to
> > recover my mtz file.
> >
> >
> > Smith
> >
> >
>
>
>


original.log
Description: Binary data


original+todos.log
Description: Binary data


original+todos+tounix.log
Description: Binary data

Re: [ccp4bb] how to recover my data

[Tim Gruene]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Dear Smith,

an mtz-file is a binary file and none of the programs you list is
suitable to open it in a meaningful manner.

Why don't you follow the advice you were already given?
mtzdump from ccp4 would be the first program to try and see if any
information can be retrieved from the mtz-file.

Regards,
Tim

On 03/06/2015 08:20 AM, Smith Lee wrote:
> Dear All, For the issue of the recovery of the mtz file, I have
> tried randomly to use excel to open one specific mtz file, however
> in this way all  the mtz files in the computer will have a excel
> icon (X), although the file extension is still .mtz. If I tried
> further to open one specific mtz file (with excel icon) with the
> notepad, all the mtz files will have the notepad icon. If I tried
> further to open one specific mtz file (with notepad icon) with the
> wordpad, all the mtz files will have the wordpad icon.  I hope
> these cluses can be helpful for you to give me the advise on
> recovery of mtz files. Smith
> 
> On Thursday, March 5, 2015 11:51 PM, Robbie Joosten
>  wrote:
> 
> 
> Hi Smith,
> 
> If this is really the problem Ian describes, you can try the Linux
> programs unix2dos and dos2unix the change the line endings. A
> potential source of the problem might be copying the file with
> certain (S)FTP clients: in 'text-mode' they change the line endings
> to your OS default to be user friendly.
> 
> Cheers, Robbie
> 
>> -Original Message- From: CCP4 bulletin board
>> [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Ian Tickle Sent:
>> Thursday, March 05, 2015 14:04 To: CCP4BB@JISCMAIL.AC.UK Subject:
>> Re: [ccp4bb] how to recover my data
>> 
>> Hi Smith
>> 
>> 
>> I sympathise with your plight - I have had to do similar things
>> in the past for other people!  I think your most fruitful course
>> of action would be to talk to the technician who recovered your
>> data because only he knows what he actually did to recover it.
>> 
>> 
>> From your description of your recovery of the PDB file it looks
>> to me like a line terminator issue, i.e. was the original file
>> created in Linux, Windows or Mac?  This is relevant because the
>> line terminators are different and it sounds like the technician
>> didn't simply copy the file, he changed the line terminators.  If
>> he did the same with the MTZ file thinking it was a text file the
>> additional line terminators would corrupt the binary data making
>> it impossible to read with any of the CCP4 MTZ utilities.  If you
>> can understand exactly what the technician did you may be able to
>> reverse it and recover the binary data.
>> 
>> 
>> Hope this helps!
>> 
>> 
>> Cheers
>> 
>> 
>> -- Ian
>> 
>> 
>> On 5 March 2015 at 05:36, Smith Lee <0459ef8548d5-dmarc- 
>> requ...@jiscmail.ac.uk> wrote:
>> 
>> 
>> 
>> Dear All,
>> 
>> Recently my computer hardware has been broken and all the data 
>> has been recovered to movable hardware by technician. However I
>> find the recovered PDB file and the MTZcould not be openned by
>> Coot. Then I open the revovered PDB file by WordPad, and from
>> WordPad I copied it to notepad and save it as pdb file. I find
>> the Coot can open the notepad saved pdb file, thus my pdb files
>> can be succesfully recovered from the hardware.
>> 
>> But will you please tell me how to have Coot open my mtz file?
>> After data recovery by the technicial, the data size of the mtz
>> file did not decrease, thus I think there is a way to have it
>> recovered.
>> 
>> I have not noticed there were similar or identical posts as mine
>> for recovery data before in the CCP4 mail list.
>> 
>> Thus I am looking forward to getting a reply from you on how to 
>> recover my mtz file.
>> 
>> 
>> Smith
>> 
>> 
> 
> 
> 

- -- 
- --
Dr Tim Gruene
Institut fuer anorganische Chemie
Tammannstr. 4
D-37077 Goettingen

GPG Key ID = A46BEE1A

-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iD8DBQFU+V2jUxlJ7aRr7hoRApjyAKCwUB2lXaXFEoFievrLgJ+YDPZg+gCcDF6x
IkBOMnu3OH/gjJknjvPWP+I=
=K7G9
-END PGP SIGNATURE-

Re: [ccp4bb] how to recover my data

[Smith Lee]

Dear All, For the issue of the recovery of the mtz file, I have tried randomly 
to use excel to open one specific mtz file, however in this way all  the mtz 
files in the computer will have a excel icon (X), although the file extension 
is still .mtz. If I tried further to open one specific mtz file (with excel 
icon) with the notepad, all the mtz files will have the notepad icon. If I 
tried further to open one specific mtz file (with notepad icon) with the 
wordpad, all the mtz files will have the wordpad icon.  I hope these cluses can 
be helpful for you to give me the advise on recovery of mtz files. Smith 

 On Thursday, March 5, 2015 11:51 PM, Robbie Joosten 
 wrote:
   

 Hi Smith,

If this is really the problem Ian describes, you can try the Linux programs 
unix2dos and dos2unix the change the line endings. A potential source of the 
problem might be copying the file with certain (S)FTP clients: in 'text-mode' 
they change the line endings to your OS default to be user friendly.

Cheers,
Robbie

> -Original Message-
> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of
> Ian Tickle
> Sent: Thursday, March 05, 2015 14:04
> To: CCP4BB@JISCMAIL.AC.UK
> Subject: Re: [ccp4bb] how to recover my data
> 
> Hi Smith
> 
> 
> I sympathise with your plight - I have had to do similar things in the past 
> for
> other people!  I think your most fruitful course of action would be to talk to
> the technician who recovered your data because only he knows what he
> actually did to recover it.
> 
> 
> From your description of your recovery of the PDB file it looks to me like a
> line terminator issue, i.e. was the original file created in Linux, Windows or
> Mac?  This is relevant because the line terminators are different and it
> sounds like the technician didn't simply copy the file, he changed the line
> terminators.  If he did the same with the MTZ file thinking it was a text file
> the additional line terminators would corrupt the binary data making it
> impossible to read with any of the CCP4 MTZ utilities.  If you can understand
> exactly what the technician did you may be able to reverse it and recover the
> binary data.
> 
> 
> Hope this helps!
> 
> 
> Cheers
> 
> 
> -- Ian
> 
> 
> On 5 March 2015 at 05:36, Smith Lee <0459ef8548d5-dmarc-
> requ...@jiscmail.ac.uk> wrote:
> 
> 
> 
>     Dear All,
> 
>     Recently my computer hardware has been broken and all the data
> has been recovered to movable hardware by technician. However I find the
> recovered PDB file and the MTZcould not be openned by Coot. Then I open
> the revovered PDB file by WordPad, and from WordPad I copied it to
> notepad and save it as pdb file. I find the Coot can open the notepad saved
> pdb file, thus my pdb files can be succesfully recovered from the hardware.
> 
>     But will you please tell me how to have Coot open my mtz file? After
> data recovery by the technicial, the data size of the mtz file did not 
> decrease,
> thus I think there is a way to have it recovered.
> 
>     I have not noticed there were similar or identical posts as mine for
> recovery data before in the CCP4 mail list.
> 
>     Thus I am looking forward to getting a reply from you on how to
> recover my mtz file.
> 
> 
>     Smith
> 
> 

   

Re: [ccp4bb] how to recover my data

[Jurgen Bosch]

Dear Smith,

Wouldn’t it be easier to reprocess your data and do a MR with your rescued PDB 
file and start from there ? Instead of trying to mess around with the 
non-readable mtz file ?

You should have those raw images somewhere, perhaps even at the beam line if 
you have not made your own copies.

Sorry for your disaster,

Jürgen

..
Jürgen Bosch
Johns Hopkins University
Bloomberg School of Public Health
Department of Biochemistry & Molecular Biology
Johns Hopkins Malaria Research Institute
615 North Wolfe Street, W8708
Baltimore, MD 21205
Office: +1-410-614-4742
Lab:  +1-410-614-4894
Fax:  +1-410-955-2926
http://lupo.jhsph.edu

On Mar 5, 2015, at 12:36 AM, Smith Lee 
<0459ef8548d5-dmarc-requ...@jiscmail.ac.uk>
 wrote:


Dear All,

Recently my computer hardware has been broken and all the data has been 
recovered to movable hardware by technician. However I find the recovered PDB 
file and the MTZcould not be openned by Coot. Then I open the revovered PDB 
file by WordPad, and from WordPad I copied it to notepad and save it as pdb 
file. I find the Coot can open the notepad saved pdb file, thus my pdb files 
can be succesfully recovered from the hardware.

But will you please tell me how to have Coot open my mtz file? After data 
recovery by the technicial, the data size of the mtz file did not decrease, 
thus I think there is a way to have it recovered.

I have not noticed there were similar or identical posts as mine for recovery 
data before in the CCP4 mail list.

Thus I am looking forward to getting a reply from you on how to recover my mtz 
file.

Smith

Re: [ccp4bb] how to recover my data

[Ian Tickle]

>
>
> MTZ files are binary, so I don't think they have line-endings in the sense
> that text files do.
>
>
> Also it's quite likely that some of the binary data in the MTZ file will
happen by chance to have the same ASCII code as the line terminator so
blindly changing or removing line terminators in the file without a
complete understanding of the algorithm by which they were introduced is
not going to work!

Cheers

-- Ian

Re: [ccp4bb] how to recover my data

[Fulvio Saccoccia, Sapienza]

Yes, sorry for the mistake;

Smith,
did you try to open your .mtz file with matzdmp from a terminal window? is it 
possible to redirect the output from mtzdmp to a new file, I mean:

>mtzdmp corrupted_file.mtz > recovered_file.mtz

???
I don’t know...

Fulvio



Il giorno 05/mar/2015, alle ore 16:18, Harry Powell  
ha scritto:

> Hi Fulvio
> 
> MTZ files are binary, so I don't think they have line-endings in the sense 
> that text files do.
> 
> On 5 Mar 2015, at 15:09, Fulvio Saccoccia, Sapienza wrote:
> 
>> Sorry, my previous post was not properly attached; I attached my idea:
>> 
>> Dear Smith,
>>It is curious; I am not sure about what I am going to suggest since I 
>> never had that problem but the simplest explanation I found is that you have 
>> some Line Endings issues. Windows, Mac and Unix all use different codes to 
>> end a line in a text file. A file that is scripted with Windows has Windows 
>> line ending (DOS-like) and may not be properly read on Mac or Linux machine. 
>> Let me explain: by using WordPad and notepad you might have changed the end 
>> line of your PDB/mtz file from UNIX-based (or Mac) line endings to 
>> Windows-based Line endings. Line endings cannot be detected by simple 
>> inspecting the file, but some programs (in this case Coot) can have problem 
>> to open them.
>> To be safe, try by opening your files with such a program (TextWrangler for 
>> ex. in Mac) and save the file with the correct line ending for your machine 
>> (Mac, Linux or Windows). I attached a picture for reference to show you the 
>> option you have in TextWrangler when you use Save as… option (other text 
>> editors should have similar options)
>> 
>> 
>> Best wishes
>> Fulvio
>> 
>> 
>> Il giorno 05/mar/2015, alle ore 14:51, Robbie Joosten 
>>  ha scritto:
>> 
>>> Hi Smith,
>>> 
>>> If this is really the problem Ian describes, you can try the Linux programs 
>>> unix2dos and dos2unix the change the line endings. A potential source of 
>>> the problem might be copying the file with certain (S)FTP clients: in 
>>> 'text-mode' they change the line endings to your OS default to be user 
>>> friendly.
>>> 
>>> Cheers,
>>> Robbie
>>> 
>>>> -Original Message-
>>>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of
>>>> Ian Tickle
>>>> Sent: Thursday, March 05, 2015 14:04
>>>> To: CCP4BB@JISCMAIL.AC.UK
>>>> Subject: Re: [ccp4bb] how to recover my data
>>>> 
>>>> Hi Smith
>>>> 
>>>> 
>>>> I sympathise with your plight - I have had to do similar things in the 
>>>> past for
>>>> other people!  I think your most fruitful course of action would be to 
>>>> talk to
>>>> the technician who recovered your data because only he knows what he
>>>> actually did to recover it.
>>>> 
>>>> 
>>>> From your description of your recovery of the PDB file it looks to me like 
>>>> a
>>>> line terminator issue, i.e. was the original file created in Linux, 
>>>> Windows or
>>>> Mac?  This is relevant because the line terminators are different and it
>>>> sounds like the technician didn't simply copy the file, he changed the line
>>>> terminators.  If he did the same with the MTZ file thinking it was a text 
>>>> file
>>>> the additional line terminators would corrupt the binary data making it
>>>> impossible to read with any of the CCP4 MTZ utilities.  If you can 
>>>> understand
>>>> exactly what the technician did you may be able to reverse it and recover 
>>>> the
>>>> binary data.
>>>> 
>>>> 
>>>> Hope this helps!
>>>> 
>>>> 
>>>> Cheers
>>>> 
>>>> 
>>>> -- Ian
>>>> 
>>>> 
>>>> On 5 March 2015 at 05:36, Smith Lee <0459ef8548d5-dmarc-
>>>> requ...@jiscmail.ac.uk> wrote:
>>>> 
>>>> 
>>>> 
>>>>Dear All,
>>>> 
>>>>Recently my computer hardware has been broken and all the data
>>>> has been recovered to movable hardware by technician. However I find the
>>>> recovered PDB file and the MTZcould not be openned by Coot. Then I open
>>>> the revovered PDB file by WordPad, and from WordPad I copied it to
>>>> notepad and save it as pdb file. I find the Coot can open the notepad saved
>>>> pdb file, thus my pdb files can be succesfully recovered from the hardware.
>>>> 
>>>>But will you please tell me how to have Coot open my mtz file? After
>>>> data recovery by the technicial, the data size of the mtz file did not 
>>>> decrease,
>>>> thus I think there is a way to have it recovered.
>>>> 
>>>>I have not noticed there were similar or identical posts as mine for
>>>> recovery data before in the CCP4 mail list.
>>>> 
>>>>Thus I am looking forward to getting a reply from you on how to
>>>> recover my mtz file.
>>>> 
>>>> 
>>>>Smith
>>>> 
>>>> 
>> 
> 
> Harry
> --
> Dr Harry Powell, MRC Laboratory of Molecular Biology, Francis Crick Avenue, 
> Cambridge Biomedical Campus, Cambridge CB2 0QH
> Chairman of International Union of Crystallography Commission on 
> Crystallographic Computing
> Chairman of European Crystallographic Association SIG9 (Crystallographic 
> Computing) 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

Re: [ccp4bb] how to recover my data

[Fulvio Saccoccia, Sapienza]

Sorry, my previous post was not properly attached; I attached my idea:

Dear Smith,
   It is curious; I am not sure about what I am going to suggest since I never 
had that problem but the simplest explanation I found is that you have some 
Line Endings issues. Windows, Mac and Unix all use different codes to end a 
line in a text file. A file that is scripted with Windows has Windows line 
ending (DOS-like) and may not be properly read on Mac or Linux machine. Let me 
explain: by using WordPad and notepad you might have changed the end line of 
your PDB/mtz file from UNIX-based (or Mac) line endings to Windows-based Line 
endings. Line endings cannot be detected by simple inspecting the file, but 
some programs (in this case Coot) can have problem to open them.
To be safe, try by opening your files with such a program (TextWrangler for ex. 
in Mac) and save the file with the correct line ending for your machine (Mac, 
Linux or Windows). I attached a picture for reference to show you the option 
you have in TextWrangler when you use Save as… option (other text editors 
should have similar options)


Best wishes
Fulvio


Il giorno 05/mar/2015, alle ore 14:51, Robbie Joosten 
 ha scritto:

> Hi Smith,
> 
> If this is really the problem Ian describes, you can try the Linux programs 
> unix2dos and dos2unix the change the line endings. A potential source of the 
> problem might be copying the file with certain (S)FTP clients: in 'text-mode' 
> they change the line endings to your OS default to be user friendly.
> 
> Cheers,
> Robbie
> 
>> -Original Message-
>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of
>> Ian Tickle
>> Sent: Thursday, March 05, 2015 14:04
>> To: CCP4BB@JISCMAIL.AC.UK
>> Subject: Re: [ccp4bb] how to recover my data
>> 
>> Hi Smith
>> 
>> 
>> I sympathise with your plight - I have had to do similar things in the past 
>> for
>> other people!  I think your most fruitful course of action would be to talk 
>> to
>> the technician who recovered your data because only he knows what he
>> actually did to recover it.
>> 
>> 
>> From your description of your recovery of the PDB file it looks to me like a
>> line terminator issue, i.e. was the original file created in Linux, Windows 
>> or
>> Mac?  This is relevant because the line terminators are different and it
>> sounds like the technician didn't simply copy the file, he changed the line
>> terminators.  If he did the same with the MTZ file thinking it was a text 
>> file
>> the additional line terminators would corrupt the binary data making it
>> impossible to read with any of the CCP4 MTZ utilities.  If you can understand
>> exactly what the technician did you may be able to reverse it and recover the
>> binary data.
>> 
>> 
>> Hope this helps!
>> 
>> 
>> Cheers
>> 
>> 
>> -- Ian
>> 
>> 
>> On 5 March 2015 at 05:36, Smith Lee <0459ef8548d5-dmarc-
>> requ...@jiscmail.ac.uk> wrote:
>> 
>> 
>> 
>>  Dear All,
>> 
>>  Recently my computer hardware has been broken and all the data
>> has been recovered to movable hardware by technician. However I find the
>> recovered PDB file and the MTZcould not be openned by Coot. Then I open
>> the revovered PDB file by WordPad, and from WordPad I copied it to
>> notepad and save it as pdb file. I find the Coot can open the notepad saved
>> pdb file, thus my pdb files can be succesfully recovered from the hardware.
>> 
>>  But will you please tell me how to have Coot open my mtz file? After
>> data recovery by the technicial, the data size of the mtz file did not 
>> decrease,
>> thus I think there is a way to have it recovered.
>> 
>>  I have not noticed there were similar or identical posts as mine for
>> recovery data before in the CCP4 mail list.
>> 
>>  Thus I am looking forward to getting a reply from you on how to
>> recover my mtz file.
>> 
>> 
>>  Smith
>> 
>> 

Re: [ccp4bb] how to recover my data

[Robbie Joosten]

Hi Smith,

If this is really the problem Ian describes, you can try the Linux programs 
unix2dos and dos2unix the change the line endings. A potential source of the 
problem might be copying the file with certain (S)FTP clients: in 'text-mode' 
they change the line endings to your OS default to be user friendly.

Cheers,
Robbie

> -Original Message-
> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of
> Ian Tickle
> Sent: Thursday, March 05, 2015 14:04
> To: CCP4BB@JISCMAIL.AC.UK
> Subject: Re: [ccp4bb] how to recover my data
> 
> Hi Smith
> 
> 
> I sympathise with your plight - I have had to do similar things in the past 
> for
> other people!  I think your most fruitful course of action would be to talk to
> the technician who recovered your data because only he knows what he
> actually did to recover it.
> 
> 
> From your description of your recovery of the PDB file it looks to me like a
> line terminator issue, i.e. was the original file created in Linux, Windows or
> Mac?  This is relevant because the line terminators are different and it
> sounds like the technician didn't simply copy the file, he changed the line
> terminators.  If he did the same with the MTZ file thinking it was a text file
> the additional line terminators would corrupt the binary data making it
> impossible to read with any of the CCP4 MTZ utilities.  If you can understand
> exactly what the technician did you may be able to reverse it and recover the
> binary data.
> 
> 
> Hope this helps!
> 
> 
> Cheers
> 
> 
> -- Ian
> 
> 
> On 5 March 2015 at 05:36, Smith Lee <0459ef8548d5-dmarc-
> requ...@jiscmail.ac.uk> wrote:
> 
> 
> 
>   Dear All,
> 
>   Recently my computer hardware has been broken and all the data
> has been recovered to movable hardware by technician. However I find the
> recovered PDB file and the MTZcould not be openned by Coot. Then I open
> the revovered PDB file by WordPad, and from WordPad I copied it to
> notepad and save it as pdb file. I find the Coot can open the notepad saved
> pdb file, thus my pdb files can be succesfully recovered from the hardware.
> 
>   But will you please tell me how to have Coot open my mtz file? After
> data recovery by the technicial, the data size of the mtz file did not 
> decrease,
> thus I think there is a way to have it recovered.
> 
>   I have not noticed there were similar or identical posts as mine for
> recovery data before in the CCP4 mail list.
> 
>   Thus I am looking forward to getting a reply from you on how to
> recover my mtz file.
> 
> 
>   Smith
> 
> 

[ccp4bb] Fwd: [ccp4bb] how to recover my data

[Fulvio Saccoccia, Sapienza]

Inizio messaggio inoltrato:

> Da: "Fulvio Saccoccia, Sapienza" 
> Oggetto: Re: [ccp4bb] how to recover my data
> Data: 05 marzo 2015 13:48:34 CET
> A: Smith Lee 
> 
> Dear Smith,
>It is curious; I am not sure about what I am going to suggest since I 
> never had that problem but the simplest explanation I found is that you have 
> some Line Endings issues. Windows, Mac and Unix all use different codes to 
> end a line in a text file. A file that is scripted with Windows has Windows 
> line ending (DOS-like) and may not be properly read on Mac or Linux machine. 
> Let me explain: by using WordPad and notepad you might have changed the end 
> line of your PDB/mtz file from UNIX-based (or Mac) line endings to 
> Windows-based Line endings. Line endings cannot be detected by simple 
> inspecting the file, but some programs (in this case Coot) can have problem 
> to open them.
> To be safe, try by opening your files with such a program (TextWrangler for 
> ex. in Mac) and save the file with the correct line ending for your machine 
> (Mac, Linux or Windows). I attached a picture for reference to show you the 
> option you have in TextWrangler when you use Save as… option (other text 
> editors should have similar options)
> 
> 
> Best wishes
> Fulvio
> 
> 
> 
> Il giorno 05/mar/2015, alle ore 06:36, Smith Lee 
> <0459ef8548d5-dmarc-requ...@jiscmail.ac.uk> ha scritto:
> 
> 
>> 
>> Dear All,
>>  
>> Recently my computer hardware has been broken and all the data has been 
>> recovered to movable hardware by technician. However I find the recovered 
>> PDB file and the MTZcould not be openned by Coot. Then I open the revovered 
>> PDB file by WordPad, and from WordPad I copied it to notepad and save it as 
>> pdb file. I find the Coot can open the notepad saved pdb file, thus my pdb 
>> files can be succesfully recovered from the hardware.
>>  
>> But will you please tell me how to have Coot open my mtz file? After data 
>> recovery by the technicial, the data size of the mtz file did not decrease, 
>> thus I think there is a way to have it recovered.
>> 
>> I have not noticed there were similar or identical posts as mine for 
>> recovery data before in the CCP4 mail list.
>>  
>> Thus I am looking forward to getting a reply from you on how to recover my 
>> mtz file.
>>  
>> Smith
> 

Re: [ccp4bb] how to recover my data

[Ian Tickle]

Hi Smith

I sympathise with your plight - I have had to do similar things in the past
for other people!  I think your most fruitful course of action would be to
talk to the technician who recovered your data because only he knows what
he actually did to recover it.

>From your description of your recovery of the PDB file it looks to me like
a line terminator issue, i.e. was the original file created in Linux,
Windows or Mac?  This is relevant because the line terminators are
different and it sounds like the technician didn't simply copy the file, he
changed the line terminators.  If he did the same with the MTZ file
thinking it was a text file the additional line terminators would corrupt
the binary data making it impossible to read with any of the CCP4 MTZ
utilities.  If you can understand exactly what the technician did you may
be able to reverse it and recover the binary data.

Hope this helps!

Cheers

-- Ian

On 5 March 2015 at 05:36, Smith Lee <
0459ef8548d5-dmarc-requ...@jiscmail.ac.uk> wrote:

>
> Dear All,
>
> Recently my computer hardware has been broken and all the data has been
> recovered to movable hardware by technician. However I find the recovered
> PDB file and the MTZcould not be openned by Coot. Then I open the revovered
> PDB file by WordPad, and from WordPad I copied it to notepad and save it as
> pdb file. I find the Coot can open the notepad saved pdb file, thus my pdb
> files can be succesfully recovered from the hardware.
>
> But will you please tell me how to have Coot open my mtz file? After data
> recovery by the technicial, the data size of the mtz file did not decrease,
> thus I think there is a way to have it recovered.
>
> I have not noticed there were similar or identical posts as mine for
> recovery data before in the CCP4 mail list.
>
> Thus I am looking forward to getting a reply from you on how to recover my
> mtz file.
>
> Smith
>

Re: [ccp4bb] how to recover my data

[Zhijie Li]

Hi Smith,

There is a viewHKL in ccp4, but that’s for viewing an intact MTZ. There are 
many tools for manipulating MTZ files in ccp4, but I doubt any of them was 
designed for you to fix a corrupt MTZ. 

You may try opening your MTZ with a Hex editor to see if its structure looks 
fine.
A Hex editor: http://www.flexhex.com/download/
(A very helpful feature of this software is that when you mouse-over a 
highlighted selection, it converts HEX numbers to DEC int or float.)


MTZ file is not very complicated. After a simple header saying “MTZ” and a 
number (multiply the int number at byte 5-8 by 4 you get the position of the 
header information), a marker and a few lines of 00, the main body of an MTZ is 
a simple data array with each data points occupying 4 bytes (a floating point 
number REAL*4). Then after this block of data, the “header information” (not 
the head of the file) are stored at the end of the file, starting with “VERS 
MTZ”.

In the data array block, since every line of data you see in view HKL starts 
with the H K L, you can easily locate the reflexion of interest by looking for 
the three simpler-looking numbers. 
For example: 
00 00 18 c2  is –38, 
00 00 00 00  is 0,
00 00 a0 40  is 5, 
So these are H K L –38 0 5 
You may also see a lot of ff fa 5a 5a in the data block. Those are unmeasured 
data I believe.

If you open your MTZ and cannot find the header information at the end then 
that means the file has lost a chunk. If you multiply the number stored at byte 
5-8 by 4 and at that address you do not see “VERS MTZ”, the files also has 
changed length, or has been partially overwritten after being deleted. 
For example, If you have 1B BC 07 00 at byte 5-8, that’s Hex 0007BC1B=506907, 
multiply by 4 you get 2027628=1EF06C. Go to address 1EF06C, the cursor should 
land on the M of “VERS MTZ”.

You can find descriptions of the MTZ format here:
http://www.ccp4.ac.uk/html/mtzformat.html
Zhijie


From: Smith Lee 
Sent: Thursday, March 05, 2015 3:46 AM
To: Zhijie Li 
Subject: Re: [ccp4bb] how to recover my data

Zhijie,

Is any mtz editor? I remember there is a software on it but I currently forget 
the name of that software.

Smith 



On Thursday, March 5, 2015 6:23 PM, Zhijie Li  wrote:




Hi Smith,

I wonder why after you copy and paste the content of PDB it became readable by 
Coot. It is possible that some sort of reading error was introduced to the PDB 
file during the recovery, which was then filtered out by the text editors 
(extended ASCII for example). What was the cause of the damage? Can you send me 
the PDB file that coot could not open? 

For the mtz files, it can be quite difficult to fix if there was a reading 
error during the recovery. Being binary certainly makes it difficult to be 
dealt with. If you still have the .sca files (if you used HKL2000) maybe 
starting from there is easier.

Zhijie



From: Smith Lee 
Sent: Thursday, March 05, 2015 12:36 AM
To: CCP4BB@JISCMAIL.AC.UK 
Subject: [ccp4bb] how to recover my data


Dear All,

Recently my computer hardware has been broken and all the data has been 
recovered to movable hardware by technician. However I find the recovered PDB 
file and the MTZcould not be openned by Coot. Then I open the revovered PDB 
file by WordPad, and from WordPad I copied it to notepad and save it as pdb 
file. I find the Coot can open the notepad saved pdb file, thus my pdb files 
can be succesfully recovered from the hardware.

But will you please tell me how to have Coot open my mtz file? After data 
recovery by the technicial, the data size of the mtz file did not decrease, 
thus I think there is a way to have it recovered.

I have not noticed there were similar or identical posts as mine for recovery 
data before in the CCP4 mail list. 

Thus I am looking forward to getting a reply from you on how to recover my mtz 
file.

Smith

[ccp4bb] how to recover my data

[Smith Lee]

Dear All, Recently my computer hardware has been broken and all the data has 
been recovered to movable hardware by technician. However I find the recovered 
PDB file and the MTZcould not be openned by Coot. Then I open the revovered PDB 
file by WordPad, and from WordPad I copied it to notepad and save it as pdb 
file. I find the Coot can open the notepad saved pdb file, thus my pdb files 
can be succesfully recovered from the hardware. But will you please tell me how 
to have Coot open my mtz file? After data recovery by the technicial, the data 
size of the mtz file did not decrease, thus I think there is a way to have it 
recovered.
I have not noticed there were similar or identical posts as mine for recovery 
data before in the CCP4 mail list.  Thus I am looking forward to getting a 
reply from you on how to recover my mtz file. Smith

[ccp4bb] How to recover my data

[Smith Lee]

Dear All, Recently my computer hardware has been broken and all the data has 
been recovered to movable hardware by technician. However I find the recovered 
PDB file and the MTZcould not be openned by Coot. Then I open the revovered PDB 
file by WordPad, and from WordPad I copied it to notepad and save it as pdb 
file. I find the Coot can open the notepad saved pdb file, thus my pdb files 
can be succesfully recovered from the hardware. But will you please tell me how 
to have Coot open my mtz file? After data recovery by the technicial, the data 
size of the mtz file did not decrease, thus I think there is a way to have it 
recovered. I am looking forward to getting a reply from you on how to recover 
my mtz file. Smith