Re: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures

[Clemens Vonrhein]

Hi Andreas,

On Thu, Mar 19, 2020 at 07:06:32AM +0100, Andreas Förster wrote:
> - The link to the raw data can be found inside the mmCIF file under
> _pdbx_related_exp_data_set.data_reference.  Whether it's also shown on one
> of the PDB sites is up to the designers of these sides.  I cannot find a
> link to the experimental data on rcsb.org.

They are available near the bottom of a rcsb.org page ("Experimental
Data & Validation" section) - see e.g. 6VS4 page

  http://www.rcsb.org/structure/6VS4

(the newest deposited PDB structure with deposited data recorded).

Cheers

Clemens

PS: if anyone is in need of a simple script for finding and
downloading such PDB entries, please let me know.



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

Re: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures

[Clemens Vonrhein]

Dear Petr,

On Thu, Mar 19, 2020 at 06:45:31AM +, Petr Kolenko wrote:
> For those of you who are in touch with these data, would deposition
> of unmerged intensities in P1 in the whole detector range instead of
> complete dataset be good enough to „correctly“ re-evaluete the
> structure? Or is there a doubt that even this approach would not
> lead to optimal structure?

It would be a step in the right direction, yes ... but:

 * It assumes that all spots have been integrated and nothing else
   than those spots: there can be cases of missed unit cell
   duplications or incorrect enforcement of a unit cell and symmetry
   (because those crystals always come in a known cell/SG, right?).

 * We'll still be stuck with a particular intergration program and a
   particular set of user decisions how to run it. Yes, all modern
   data processing programs and packages are usually doing a great job
   if run in (more or less) default mode on (more or less) good
   crystals. But not everything is lysozyme/thaumatin/XYZ collected
   fine-sliced on a PAD with low-dose, high-multiplicity on a well
   calibrated instrument ;-)

 * If integration was done in P1 the cell parameter restraints of the
   (potentiually different) true SG were not enforced - which can lead
   to incorrect cell parameters (which has knock-on effects in
   refinement ... remember the WhatIf check for correct cell
   parameter?).

   If integration was done in a non-P1 SG and that choice was
   incorrect: how do we know that those cell parameters didn't enforce
   an incorrect set of cell parameter restraints, leading to poor
   integration? Again: different programs have different ways of
   assigning pixels (and their counts) to a reflection intensity
   (concepts like shoebox, profile-fitting, pixel-labeling etc all
   come to mind).

 * Integration itself can be suboptimal if there are issues with the
   crystal (ice rings, split, multiple lattices etc), the instrument
   (damaged pixels, beam jitter, incorrect countrate cut-off handling
   etc) or the experiment (radiation damage etc). Data processing
   defaults might not always work correctly/best for all cases.

 * The raw images allow such corrections (damaged pixels not yet in
   pixel mask, incorrect handling of countrate_cutoff) and additional
   analysis (e.g. radiation damage analysis a la F(early)-F(late) maps
   [1]).

So the raw integrated (not scaled/corrected) and unmerged intensities
might indeed often be good enough, but if we are going to try and
deposit more than just the merged and scaled intensities/amplitudes
(as we do now) - and this will involve additional work and change from
our side - then why not bite the bullet once and try to go all the way
to raw images:

  * The infrastructure is already in place (proteindiffraction.org,
data.sbgrid.org, zenodo.org, wwPDB DOIs, etc), ready for use right
now.

  * From experience at multiple worshops: everyone is comfortable to
start data processing from raw images using the various software
tools out there that people are familiar with (while jumping in
half way is probably more challenging to non power-users).

Anyway, just some ideas - from someone downloading nearly all deposited
raw image data on a regular basis for testing and developing data
processing software (and finding a lot of issues in our software this
way ... as well as issues with experiments and sometimes the
instrument).

Cheers

Clemens

[1] as e.g. done in autoPROC+BUSTER



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

Re: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures

[Andreas Förster]

Hi Clemens,

this is curious.  You are right that there's a direct link to the raw data
on 6vs4, but there isn't one on http://www.rcsb.org/structure/6H56.  I'll
contact RCSB to see where the problem lies.

All best.


Andreas



On Thu, Mar 19, 2020 at 8:34 AM Clemens Vonrhein 
wrote:

> Hi Andreas,
>
> On Thu, Mar 19, 2020 at 07:06:32AM +0100, Andreas Förster wrote:
> > - The link to the raw data can be found inside the mmCIF file under
> > _pdbx_related_exp_data_set.data_reference.  Whether it's also shown on
> one
> > of the PDB sites is up to the designers of these sides.  I cannot find a
> > link to the experimental data on rcsb.org.
>
> They are available near the bottom of a rcsb.org page ("Experimental
> Data & Validation" section) - see e.g. 6VS4 page
>
>   http://www.rcsb.org/structure/6VS4
>
> (the newest deposited PDB structure with deposited data recorded).
>
> Cheers
>
> Clemens
>
> PS: if anyone is in need of a simple script for finding and
> downloading such PDB entries, please let me know.
>
>



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

Re: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures

[Julien Cappèle]

Hi everyone,

There are some very interesting ideas. 

Though I agree with you Clement that raw images are amazing to work with as you 
can use any software you are confortable with, we cannot forget that depositing 
several TB of data for each lab would be bad for ecological reason. And because 
detectors are always improving (thank you all!), size of data will increase 
exponentially. 

Correct me if I'm wrong, as I am not that familiar with the way that 
integration of raw images works:

Could it be possible for a new/already existing software to store reflections 
(area, intensity from center to border, position x/y on the image, and 
information of the image) in a lightweight and text only file ? Possibly a new 
format to be used for integration ?

While those text files would be heavy, they'd be still lighter than raw images 
and the whole useless white space they carry with them between reflections.

If not possible, could we imagine to eliminate all the unused space on these ? 
Like a super raw image diet-plan for the incoming summer !

Best regards,

Julien CAPPELE
Université de Lorraine, Nancy, France
PhD student - 2nd Year



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

Re: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures

[Winter, Graeme (DLSLtd,RAL,LSCI)]

Hi Folks,

Quick note on "Like a super raw image diet-plan for the incoming summer” - the 
bslz4 compression used with Eiger works really well, and the data are pretty 
close to entropy limit in terms of size - which means if you measure your data 
carefully (i.e. low background etc.) you can realistically have a full Eiger 
data set which will fit on a DVD -

Grey-Area i04-ins :) $ du -ach insu_d200_1*h5
2.5Ginsu_d200_1_01.h5
1.9Ginsu_d200_1_02.h5
 92Kinsu_d200_1_master.h5
138Minsu_d200_1_meta.h5
4.0Kinsu_d200_1_meta_pack.h5
4.5Gtotal

which I think is pretty modest storage wise - this is 1,800 image Eiger 16M 
data set so pretty realistic. 

No matter how much data we produce, the particle physicists have us beat, and I 
would wager that netflix is a couple of orders of magnitude worse than MX for 
the environment. 

Also, it costs a lot more CO2 to produce the data than to store it - 
synchrotrons don’t run on wind turbines ;-)  

I agree with Clemens - if we are serious about revisiting the data, you have to 
have access to the actual data rather than a description of the data (which is 
essentially what an MTZ file is) 

A small administrative note for those uploading to zenodo etc - 

if you are uploading HDF5 as above, this works fine
if you are uploading CBF images, would suggest you gzip / bzip2 compress them 
(pick one) and then tar up each sweep independently before upload

 - makes fetching the data back down again efficient. Zenodo does not do well 
with many many small files (I have discussed this with the zenodo developers - 
it is a reasonable design choice) 

Finally I would ask if you do upload raw data please also upload the beamline 
parameters you used e.g. the beam centre from a white board or whatever - if 
it’s not in the headers it is not known 

All the best Graeme

> On 19 Mar 2020, at 08:47, Julien Cappèle  
> wrote:
> 
> Hi everyone,
> 
> There are some very interesting ideas. 
> 
> Though I agree with you Clement that raw images are amazing to work with as 
> you can use any software you are confortable with, we cannot forget that 
> depositing several TB of data for each lab would be bad for ecological 
> reason. And because detectors are always improving (thank you all!), size of 
> data will increase exponentially. 
> 
> Correct me if I'm wrong, as I am not that familiar with the way that 
> integration of raw images works:
> 
> Could it be possible for a new/already existing software to store reflections 
> (area, intensity from center to border, position x/y on the image, and 
> information of the image) in a lightweight and text only file ? Possibly a 
> new format to be used for integration ?
> 
> While those text files would be heavy, they'd be still lighter than raw 
> images and the whole useless white space they carry with them between 
> reflections.
> 
> If not possible, could we imagine to eliminate all the unused space on these 
> ? Like a super raw image diet-plan for the incoming summer !
> 
> Best regards,
> 
> Julien CAPPELE
> Université de Lorraine, Nancy, France
> PhD student - 2nd Year
> 
> 
> 
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1


-- 
This e-mail and any attachments may contain confidential, copyright and or 
privileged material, and are for the use of the intended addressee only. If you 
are not the intended addressee or an authorised recipient of the addressee 
please notify us of receipt by returning the e-mail and do not use, copy, 
retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not 
necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments 
are free from viruses and we cannot accept liability for any damage which you 
may sustain as a result of software viruses which may be transmitted in or with 
the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and 
Wales with its registered office at Diamond House, Harwell Science and 
Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

Re: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures

[Kroon-Batenburg, L.M.J. (Loes)]

Dear Gerard,

This is a great idea. Of course I am very much in favour of making available 
raw diffraction images, and such a virtual workshop could demonstrate the 
usefulness of reprocessing raw diffraction data and structural refinements. I 
am not at all afraid that archiving of raw data that are the basis of a 
scientific paper will have significant environmental effects: this is minor 
compared to our everyday use of cloud services.  And as Graeme mentioned: when 
archiving raw data make sure to add sufficient and correct meta data.

Best wishes,
Loes


___

Dr. Loes Kroon-Batenburg

Dept. of Crystal and Structural Chemistry
Bijvoet Center for Biomolecular Research
Utrecht University
Padualaan 8, 3584 CH Utrecht
The Netherlands

E-mail : l.m.j.kroon-batenb...@uu.nl
phone  : +31-30-2532865
fax: +31-30-2533940


Van: CCP4 bulletin board  namens Gerard Bricogne 

Verzonden: woensdag 18 maart 2020 23:30
Aan: CCP4BB@JISCMAIL.AC.UK 
Onderwerp: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures

Dear colleagues,

Perusal and some initial (re-)refinement of the various SARS-CoV-2 protease
structures in the PDB seems to indicate that that there might be potential
to improve these if refinements could be repeated after some reprocessing
and further analysis of the raw diffraction images, rather than against the
deposited merged data. This statement should in no way be construed as a
criticism of the remarkable achievements of the research groups concerned,
who have been operating under tremendous time pressure, but as an exciting
opportunity to push methods to their limits on a uniquely significant class
of structures.

Another consideration is that the various logistical problems created by
COVID-19 may soon make it increasingly difficult to collect new diffraction
data on potential drug targets relevant to the fight against SARS-CoV-2,
underlining the importance of ensuring that the best results be obtained
from every dataset actually collected, and that the most useful conclusions
be drawn from the analysis of those datasets towards improving the quality
of subsequent data collections.

On this basis we would like to propose that special efforts be made to grant
public access to the raw image data associated with any SARS-CoV-2 related
structure that is deposited into the PDB. This can be done by (1) archiving
these raw image data using resources such as data.sbgrid.org, zenodo.org,
proteindiffraction.org or any other cloud-based data-sharing service, and
(2) communicating the corresponding DOIs to the wwPDB centres. This idea
could be extended to datasets that investigators would like to offer to
interested methods developers or expert users at the pre-deposition stage.

Experts making use of those raw data would be encouraged to document, in as
much detail as possible, how particular programs or workflows could be used
on those structures/datasets to obtain the best results. This would be a
kind of "virtual workshop", a particularly valuable collective activity at
the present time when several in-person workshops (e.g. RapiData) have been
cancelled and many meetings are in limbo for several months.

The latter activity would benefit from having a centralised facility set up
for the experts to post their results and annotations: we could create such
a facility, but other, larger groups might want to consider doing so.


With best wishes,

Clemens & Gerard.



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

Re: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures

[Peter Keller]

Dear all,

On Thu, 19 Mar 2020, Winter, Graeme (DLSLtd,RAL,LSCI) wrote:


Date: Thu, 19 Mar 2020 09:08:21
From: "Winter, Graeme (DLSLtd,RAL,LSCI)" 
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures



No matter how much data we produce, the particle physicists have us beat, 
and I would wager that netflix is a couple of orders of magnitude worse 
than MX for the environment.


And not only the particle physicists - the astronomers too. A few years ago, 
I came across a figure for the daily data output rate of one of the projects 
that scans observatory data for objects that might be on a collision course 
with the Earth. We are a drop in the ocean in comparison.


Anyway, whatever the balance of opinion on long-term archiving of raw image 
data, that is not the issue that Gerard and Clemens have raised. It is 
rather making raw data available to the maximum number of researchers on a 
massively important and urgent question of pubic health. It is in the same 
spirit as the sequence of SARS-CoV-2 having been made immediately available 
by the Chinese researchers who first obtained it. I think that the case for 
doing so where there are no major technical impediments is unanswerable.


Regards,
Peter.


--
Peter Keller Tel.: +44 (0)1223 353033
Global Phasing Ltd., Fax.: +44 (0)1223 366889
Sheraton House,
Castle Park,
Cambridge CB3 0AX
United Kingdom



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

Re: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures

[Clemens Vonrhein]

Dear Julien,

On Thu, Mar 19, 2020 at 08:47:20AM +, Julien Cappèle wrote:
> Though I agree with you Clemens that raw images are amazing to work
> with as you can use any software you are confortable with, we cannot
> forget that depositing several TB of data for each lab would be bad
> for ecological reason.

Of course, there are ecological (carbon footprint) considerations -
and there are lots of papers and studies about that. I haven't looked
at any numbers, but maybe some points:

 * A lot of data is already stored (e.g. at synchrotrons) and would
   "only" needed to be made "visible" via a DOI (caveat: I realise
   that there are huge technical issues with that)

 * How does that energy consumption compare with the energy used to
   perform the experiment in the first place?

 * If by having that data available we can improve software and the
   way experiments are done: wouldn't that potentially save energy in
   hte long run (avoiding poor or unnecessary experiments in the first
   place)?

 * We are looking at a move to increase the number of raw image data
   depositions for deposited PDB structures - not at a requirement to
   deposit raw images for every PDB structure or even for every
   dataset ever collected.

   At the moment there are about 4500 image datasets available for
   about 10 PDB X-Ray structures, i.e. ~5%. 

> And because detectors are always improving (thank you all!), size of
> data will increase exponentially.

True ... and some type of experiment can benefit from those larger,
faster and more numerous types of datasets - if done correctly.

> Could it be possible for a new/already existing software to store
> reflections (area, intensity from center to border, position x/y on
> the image, and information of the image) in a lightweight and text
> only file ? Possibly a new format to be used for integration ?

See my other reply: this all assumes that the initial processing step
caught all spots (and nothing else) on the 2D image correctly.

There have been all kind of initiatives about raw data deposition (in
no particular order)

  https://www.iucr.org/resources/data/dddwg
  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5331468/
  https://www.sciencedaily.com/releases/2016/11/161108130045.htm
  https://journals.iucr.org/d/issues/2016/11/00/yt5099/
  https://onlinelibrary.wiley.com/iucr/doi/10.1107/S0909049513020724
  http://scripts.iucr.org/cgi-bin/paper?S0907444908015540
  https://scripts.iucr.org/cgi-bin/paper?dz5309
  https://bl831.als.lbl.gov/~jamesh/lossy_compression/

So we've been there before. Let's see if we can't do at least
something for the clearly important structures and work right now -
and worry about some long-term impact later (having maybe learned
something along the way). Just because we could be doing something now
doesn't mean we will have to keep doing this in a 1-N years time,
right ;-)

Cheers

Clemens



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

Re: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures

[Joel Sussman]

  19-Mar-2020
Dear Loes, Peter, Clemens & Gerard,
I concur that it is crucial to preserve the original diffraction data and make 
it available to anyone who would like to use it.
As an example, please see the very recent paper by
Nachon et al (2020). "A second look at the crystal structures of Drosophila 
melanogaster acetylcholinesterase in complex with tacrine derivatives provides 
Insights concerning catalytic intermediates and the design of specific 
insecticides" Molecules 25 pii: E1198
[https://www.ncbi.nlm.nih.gov/pubmed/32155891].
The study reexamines the original data, with modern software tools, the 
original data of a paper we published in 2000 (~20 years ago) and revealed 
features that had not been noticed. Specifically
1) previously unmodeled density in the native active site can be interpreted as 
stable acetylation of the catalytic serine.
2) Similarly, a strong density in the DmAChE/ZA complex, originally attributed 
to a sulfate ion, is better interpreted as a small molecule that is covalently 
bound. The complex is reminiscent of the carboxylate/BChE complexes observed in 
crystal structures of hBChE [Brazzolotto et al, 2012; Nicolet et al, 2003], and 
demonstrates the remarkable ability of ChEs to stabilize covalent complexes 
with carboxylates.
Thus, the study demonstrates that updated processing of older diffraction 
images, and the re-refinement of older diffraction data, can produce valuable 
information that could not be detected in the original analysis, and strongly 
supports the preservation of the diffraction images in public data banks.
Best regards
Joel

Prof. Joel L. Sussman.
joel.suss...@weizmann.ac.il   
www.weizmann.ac.il/~joel
Dept. of Structural Biology   tel: +972  (8) 934 6309   
proteopedia.org
Weizmann Institute of Science fax: +972  (8) 934 6312
Rehovot 76100 ISRAEL  mob: +972 (50) 510 9600
-

On 19 Mar 2020, at 11:32, Kroon-Batenburg, L.M.J. (Loes) 
mailto:l.m.j.kroon-batenb...@uu.nl>> wrote:

Dear Gerard,

This is a great idea. Of course I am very much in favour of making available 
raw diffraction images, and such a virtual workshop could demonstrate the 
usefulness of reprocessing raw diffraction data and structural refinements. I 
am not at all afraid that archiving of raw data that are the basis of a 
scientific paper will have significant environmental effects: this is minor 
compared to our everyday use of cloud services.  And as Graeme mentioned: when 
archiving raw data make sure to add sufficient and correct meta data.

Best wishes,
Loes

___
Dr. Loes Kroon-Batenburg
Dept. of Crystal and Structural Chemistry
Bijvoet Center for Biomolecular Research
Utrecht University
Padualaan 8, 3584 CH Utrecht
The Netherlands

E-mail : l.m.j.kroon-batenb...@uu.nl
phone  : +31-30-2532865
fax: +31-30-2533940


Van: CCP4 bulletin board mailto:CCP4BB@JISCMAIL.AC.UK>> 
namens Gerard Bricogne mailto:g...@globalphasing.com>>
Verzonden: woensdag 18 maart 2020 23:30
Aan: CCP4BB@JISCMAIL.AC.UK 
mailto:CCP4BB@JISCMAIL.AC.UK>>
Onderwerp: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures

Dear colleagues,

Perusal and some initial (re-)refinement of the various SARS-CoV-2 protease
structures in the PDB seems to indicate that that there might be potential
to improve these if refinements could be repeated after some reprocessing
and further analysis of the raw diffraction images, rather than against the
deposited merged data. This statement should in no way be construed as a
criticism of the remarkable achievements of the research groups concerned,
who have been operating under tremendous time pressure, but as an exciting
opportunity to push methods to their limits on a uniquely significant class
of structures.

Another consideration is that the various logistical problems created by
COVID-19 may soon make it increasingly difficult to collect new diffraction
data on potential drug targets relevant to the fight against SARS-CoV-2,
underlining the importance of ensuring that the best results be obtained
from every dataset actually collected, and that the most useful conclusions
be drawn from the analysis of those datasets towards improving the quality
of subsequent data collections.

On this basis we would like to propose that special efforts be made to grant
public access to the raw image data associated with any SARS-CoV-2 related
structure that is deposited into the PDB. This can be done by (1) archiving
these raw image data using resources such as 
data.sbgrid.org, zenodo.org,
proteindiffrac

Re: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures

[John Berrisford]

Dear all

The wwPDB OneDep system allows depositors to provide DOIs of raw
diffraction images during deposition to the PDB and once again
encourages depositors to provide a DOI for raw images when they have
submitted.  

Out of the 9665 X-ray entries that were released in 2019 we have DOI's
for raw images in 205 of these entries.  

We would encourage depositors to provide the DOI for their raw images
when they are available.  

Regards

John


On Mar 19 2020, at 9:48 am, Joel Sussman  wrote:

> 19-Mar-2020
> Dear Loes, Peter, Clemens & Gerard,
> I concur that it is crucial to preserve the original diffraction data
> and make it available to anyone who would like to use it.
> As an example, please see the very recent paper by 
> Nachon et al (2020). "A second look at the crystal structures of
> Drosophila melanogaster acetylcholinesterase in complex with tacrine
> derivatives provides Insights concerning catalytic intermediates and
> the design of specific insecticides" Molecules 25 pii: E1198 
> [https://www.ncbi.nlm.nih.gov/pubmed/32155891].
> The study reexamines the original data, with modern software tools,
> the original data of a paper we published in 2000 (~20 years ago) and
> revealed features that had not been noticed. Specifically 
> 1) previously unmodeled density in the native active site can be
> interpreted as stable acetylation of the catalytic serine. 
> 2) Similarly, a strong density in the DmAChE/ZA complex, originally
> attributed to a sulfate ion, is better interpreted as a small molecule
> that is covalently bound. The complex is reminiscent of the
> carboxylate/BChE complexes observed in crystal structures of hBChE
> [Brazzolotto et al, 2012; Nicolet et al, 2003], and demonstrates the
> remarkable ability of ChEs to stabilize covalent complexes with carboxylates.
> Thus, the study demonstrates that updated processing of older
> diffraction images, and the re-refinement of older diffraction data,
> can produce valuable information that could not be detected in the
> original analysis, and strongly supports the preservation of the
> diffraction images in public data banks.
> Best regards
> Joel
> 
> Prof. Joel L. Sussman.        joel.suss...@weizmann.ac.il   
> www.weizmann.ac.il/~joel
> Dept. of Structural Biology   tel: +972  (8) 934 6309       proteopedia.org
> Weizmann Institute of Science fax: +972  (8) 934 6312
> Rehovot 76100 ISRAEL          mob: +972 (50) 510 9600
> -
>  
>  
>> On 19 Mar 2020, at 11:32, Kroon-Batenburg, L.M.J. (Loes)
>>  wrote:
>>  
>> Dear Gerard,
>>  
>> This is a great idea. Of course I am very much in favour of making
>> available raw diffraction images, and such a virtual workshop could
>> demonstrate the usefulness of reprocessing raw diffraction data and
>> structural refinements. I am not at all afraid that archiving of raw
>> data that are the basis of a scientific paper will have significant
>> environmental effects: this is minor compared to our everyday use of
>> cloud services.  And as Graeme mentioned: when archiving raw data
>> make sure to add sufficient and correct meta data.
>>  
>> Best wishes,
>> Loes
>>  
>> ___
>> Dr. Loes Kroon-Batenburg
>> Dept. of Crystal and Structural Chemistry
>> Bijvoet Center for Biomolecular Research
>> Utrecht University
>> Padualaan 8, 3584 CH Utrecht
>> The Netherlands
>>  
>> E-mail : l.m.j.kroon-batenb...@uu.nl
>> phone  : +31-30-2532865
>> fax    : +31-30-2533940
>>  
>> Van: CCP4 bulletin board  namens Gerard
>> Bricogne 
>> Verzonden: woensdag 18 maart 2020 23:30
>> Aan: CCP4BB@JISCMAIL.AC.UK 
>> Onderwerp: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures
>>  
>> Dear colleagues,
>>  
>> Perusal and some initial (re-)refinement of the various SARS-CoV-2 protease
>> structures in the PDB seems to indicate that that there might be potential
>> to improve these if refinements could be repeated after some reprocessing
>> and further analysis of the raw diffraction images, rather than
>> against the
>> deposited merged data. This statement should in no way be construed
>> as a
>> criticism of the remarkable achievements of the research groups concerned,
>> who have been operating under tremendous time pressure, but as an exciting
>> opportunity to push methods to their limits on a uniquely significant class
>> of structures.
>>  
>> Another consideration is that the various logistical problems created by
>> COVID-19 may soon make it increasingly difficult to collect new diffraction
>> data on potential drug targets relevant to the fight against SARS-CoV-2,
>> underlining the importance of ensuring that the best results be obtained
>> from every dataset actually collected, and that the most useful conclusions
>> be drawn from the analysis of those datasets towards improving 

Re: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures

[Andreas Förster]

This last, very important point made by Graeme gives me the chance to plug
a satellite workshop at the IUCr Congress (end of August, so fingers
crossed that this will go ahead) on "MX raw image data formats, metadata
and validation".  This is organized by Herbert Bernstein, Loes
Kroon-Batenburg, Brian McMahon and myself, and will take place on 22 Aug
from morning till 3 pm.

Why is this relevant?  Over the past few years, many beamline scientists
from synchrotrons (and XFELs) the world over have worked hard to turn the
perception of a lack of metadata into a transcription of essential metadata
items in a language that can be saved with current data formats (HDF5 and
CBF), can be used for archiving and retrieval of data, and is understood by
automatic processing pipelines.  We like to call this the Gold Standard.
The Gold Standard demands the beam center is recorded for every
experiment.  It also insists on synchrotron and beamline, items that are
curiously absent from a lot of datasets.  In the Gold Standard, you'll find
a full description of the diffraction experiment and won't have to guess
anymore what the rotation axis is and how the detector is positioned.

The purpose of the Prague workshop is to make the Gold Standard better
known to a wider audience, explain what's in it and how different
synchrotrons are applying it, and discuss how to expand this to chemical
crystallography.  The program is still a work in progress and the website
rather austere (
https://www.iucr.org/resources/data/commdat/prague-workshop-mx-raw-data),
but mark your calendars if you're interested.  We'll have a great line-up
of speakers and ample opportunity for discussions.  As the workshop will
take place in the conference center, it will be only a short walk to the
opening ceremony later that day.

All best.


Andreas (wearing my DECTRIS hat).



On Thu, Mar 19, 2020 at 10:08 AM Winter, Graeme (DLSLtd,RAL,LSCI) <
graeme.win...@diamond.ac.uk> wrote:

> Hi Folks,
>
> Quick note on "Like a super raw image diet-plan for the incoming summer” -
> the bslz4 compression used with Eiger works really well, and the data are
> pretty close to entropy limit in terms of size - which means if you measure
> your data carefully (i.e. low background etc.) you can realistically have a
> full Eiger data set which will fit on a DVD -
>
> Grey-Area i04-ins :) $ du -ach insu_d200_1*h5
> 2.5Ginsu_d200_1_01.h5
> 1.9Ginsu_d200_1_02.h5
>  92Kinsu_d200_1_master.h5
> 138Minsu_d200_1_meta.h5
> 4.0Kinsu_d200_1_meta_pack.h5
> 4.5Gtotal
>
> which I think is pretty modest storage wise - this is 1,800 image Eiger
> 16M data set so pretty realistic.
>
> No matter how much data we produce, the particle physicists have us beat,
> and I would wager that netflix is a couple of orders of magnitude worse
> than MX for the environment.
>
> Also, it costs a lot more CO2 to produce the data than to store it -
> synchrotrons don’t run on wind turbines ;-)
>
> I agree with Clemens - if we are serious about revisiting the data, you
> have to have access to the actual data rather than a description of the
> data (which is essentially what an MTZ file is)
>
> A small administrative note for those uploading to zenodo etc -
>
> if you are uploading HDF5 as above, this works fine
> if you are uploading CBF images, would suggest you gzip / bzip2 compress
> them (pick one) and then tar up each sweep independently before upload
>
>  - makes fetching the data back down again efficient. Zenodo does not do
> well with many many small files (I have discussed this with the zenodo
> developers - it is a reasonable design choice)
>
> Finally I would ask if you do upload raw data please also upload the
> beamline parameters you used e.g. the beam centre from a white board or
> whatever - if it’s not in the headers it is not known
>
> All the best Graeme
>
> > On 19 Mar 2020, at 08:47, Julien Cappèle <
> julien.capp...@univ-lorraine.fr> wrote:
> >
> > Hi everyone,
> >
> > There are some very interesting ideas.
> >
> > Though I agree with you Clement that raw images are amazing to work with
> as you can use any software you are confortable with, we cannot forget that
> depositing several TB of data for each lab would be bad for ecological
> reason. And because detectors are always improving (thank you all!), size
> of data will increase exponentially.
> >
> > Correct me if I'm wrong, as I am not that familiar with the way that
> integration of raw images works:
> >
> > Could it be possible for a new/already existing software to store
> reflections (area, intensity from center to border, position x/y on the
> image, and information of the image) in a lightweight and text only file ?
> Possibly a new format to be used for integration ?
> >
> > While those text files would be heavy, they'd be still lighter than raw
> images and the whole useless white space they carry with them between
> reflections.
> >
> > If not possible, could we imagine to eliminate all the

Re: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures

[Harry Powell - CCP4BB]

Hi

> While those text files would be heavy, they'd be still lighter than raw 
> images and the whole useless white space they carry with them between 
> reflections.

At the risk of extending this thread into a different direction, the "white 
space (the images) carry with them between reflections” is only useless because 
we (the developers of integration software and structural biologists who use 
X-ray diffraction) choose to throw it away. The same as the can that surrounds 
my beans in tomato sauce...

There are plenty of people out there who have proposed using the non-Bragg 
scattering (aka “diffuse scattering”, inter alia) to give extra structural 
information.

My two ha’porth…

Harry

> On 19 Mar 2020, at 08:47, Julien Cappèle  
> wrote:
> 
> Hi everyone,
> 
> There are some very interesting ideas. 
> 
> Though I agree with you Clement that raw images are amazing to work with as 
> you can use any software you are confortable with, we cannot forget that 
> depositing several TB of data for each lab would be bad for ecological 
> reason. And because detectors are always improving (thank you all!), size of 
> data will increase exponentially. 
> 
> Correct me if I'm wrong, as I am not that familiar with the way that 
> integration of raw images works:
> 
> Could it be possible for a new/already existing software to store reflections 
> (area, intensity from center to border, position x/y on the image, and 
> information of the image) in a lightweight and text only file ? Possibly a 
> new format to be used for integration ?
> 
> While those text files would be heavy, they'd be still lighter than raw 
> images and the whole useless white space they carry with them between 
> reflections.
> 
> If not possible, could we imagine to eliminate all the unused space on these 
> ? Like a super raw image diet-plan for the incoming summer !
> 
> Best regards,
> 
> Julien CAPPELE
> Université de Lorraine, Nancy, France
> PhD student - 2nd Year
> 
> 
> 
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

Re: [ccp4bb] Raw diffraction images for SARS-CoV-2 related structures

[Filipe Maia]

Dear Colleagues,

I would like to add my voice to those advocating making available raw
diffraction images and point out another resource for raw data deposition,
https://cxidb.org, particularly for the case of emerging methods such as
serial crystallography.

Cheers,
Filipe

On Thu, 19 Mar 2020 at 09:01, Clemens Vonrhein 
wrote:

> Dear Petr,
>
> On Thu, Mar 19, 2020 at 06:45:31AM +, Petr Kolenko wrote:
> > For those of you who are in touch with these data, would deposition
> > of unmerged intensities in P1 in the whole detector range instead of
> > complete dataset be good enough to „correctly“ re-evaluete the
> > structure? Or is there a doubt that even this approach would not
> > lead to optimal structure?
>
> It would be a step in the right direction, yes ... but:
>
>  * It assumes that all spots have been integrated and nothing else
>than those spots: there can be cases of missed unit cell
>duplications or incorrect enforcement of a unit cell and symmetry
>(because those crystals always come in a known cell/SG, right?).
>
>  * We'll still be stuck with a particular intergration program and a
>particular set of user decisions how to run it. Yes, all modern
>data processing programs and packages are usually doing a great job
>if run in (more or less) default mode on (more or less) good
>crystals. But not everything is lysozyme/thaumatin/XYZ collected
>fine-sliced on a PAD with low-dose, high-multiplicity on a well
>calibrated instrument ;-)
>
>  * If integration was done in P1 the cell parameter restraints of the
>(potentiually different) true SG were not enforced - which can lead
>to incorrect cell parameters (which has knock-on effects in
>refinement ... remember the WhatIf check for correct cell
>parameter?).
>
>If integration was done in a non-P1 SG and that choice was
>incorrect: how do we know that those cell parameters didn't enforce
>an incorrect set of cell parameter restraints, leading to poor
>integration? Again: different programs have different ways of
>assigning pixels (and their counts) to a reflection intensity
>(concepts like shoebox, profile-fitting, pixel-labeling etc all
>come to mind).
>
>  * Integration itself can be suboptimal if there are issues with the
>crystal (ice rings, split, multiple lattices etc), the instrument
>(damaged pixels, beam jitter, incorrect countrate cut-off handling
>etc) or the experiment (radiation damage etc). Data processing
>defaults might not always work correctly/best for all cases.
>
>  * The raw images allow such corrections (damaged pixels not yet in
>pixel mask, incorrect handling of countrate_cutoff) and additional
>analysis (e.g. radiation damage analysis a la F(early)-F(late) maps
>[1]).
>
> So the raw integrated (not scaled/corrected) and unmerged intensities
> might indeed often be good enough, but if we are going to try and
> deposit more than just the merged and scaled intensities/amplitudes
> (as we do now) - and this will involve additional work and change from
> our side - then why not bite the bullet once and try to go all the way
> to raw images:
>
>   * The infrastructure is already in place (proteindiffraction.org,
> data.sbgrid.org, zenodo.org, wwPDB DOIs, etc), ready for use right
> now.
>
>   * From experience at multiple worshops: everyone is comfortable to
> start data processing from raw images using the various software
> tools out there that people are familiar with (while jumping in
> half way is probably more challenging to non power-users).
>
> Anyway, just some ideas - from someone downloading nearly all deposited
> raw image data on a regular basis for testing and developing data
> processing software (and finding a lot of issues in our software this
> way ... as well as issues with experiments and sometimes the
> instrument).
>
> Cheers
>
> Clemens
>
> [1] as e.g. done in autoPROC+BUSTER
>
> 
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1
>



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

Re: [ccp4bb] Coot under WSL2

[Paul Emsley]

On 19/03/2020 12:43, Clement Degut wrote:


I have been recently trying to use ccp4 in windows subsystem for Linux 2 (Ubuntu distribution), everything i 
tried run perfectly smoothly, including graphical interfaces.


I am pleasantly surprised - when I tried that (some time ago) the graphics performance as terrible. Perhaps 
you have better hardware.


My only problem is with coot (latest build), the interface is working fine, as well as the 3D graphical. The 
only problem is that the graphical window is offset from the rest of the UI (see screenshot).

So although it looks like everything would work smoothly it's not really usable.


What, would you say, is the biggest problem?

I guess it comes from the absence of GPU access under WSL2 and so no proper display driver, but does anyone 
had this problem? (and managed to solve it).


FWIW, I have not seen this problem before (and don't know how to fix it)

It seems to me an X server problem (or configuration issue). Have you tweaked or disabled the native OpenGL 
options on the X server?



Would be very useful for people working from home without a full linux dual 
boot.


Hmm.

In a more general note, it seems that WSL2 will be able to replace windows specific build very effectively 
in the very near future.


For everything other than OpenGL graphics (and maybe the Cloud would also be 
problematic), yes, it seems so.

Paul.



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

Re: [ccp4bb] Coot under WSL2

[Clement Degut]

The performance are massively improved in WSL2 versus WSL1, so maybe it's
why you had bad memory of it ?
WSL2 is currently only available in insider fast ring, but will be for
everyone with next major update (v2004) in few weeks.

The main problem is, while you can use the right toolbar by simply
detaching it, selecting residues is almost impossible as the pointer is
still using the main UI as reference, but the 3D display  is away, so you
need to click far from the residus you try to select.

I am playing around with the xserver options, but am not sure what I am
doing with that windows X server.
I'll update if I find a solution.

Clément

On Thu, 19 Mar 2020 at 13:31, Paul Emsley  wrote:

> On 19/03/2020 12:43, Clement Degut wrote:
> >
> > I have been recently trying to use ccp4 in windows subsystem for Linux 2
> (Ubuntu distribution), everything i
> > tried run perfectly smoothly, including graphical interfaces.
>
> I am pleasantly surprised - when I tried that (some time ago) the graphics
> performance as terrible. Perhaps
> you have better hardware.
>
> > My only problem is with coot (latest build), the interface is working
> fine, as well as the 3D graphical. The
> > only problem is that the graphical window is offset from the rest of the
> UI (see screenshot).
> > So although it looks like everything would work smoothly it's not really
> usable.
>
> What, would you say, is the biggest problem?
>
> > I guess it comes from the absence of GPU access under WSL2 and so no
> proper display driver, but does anyone
> > had this problem? (and managed to solve it).
>
> FWIW, I have not seen this problem before (and don't know how to fix it)
>
> It seems to me an X server problem (or configuration issue). Have you
> tweaked or disabled the native OpenGL
> options on the X server?
>
> > Would be very useful for people working from home without a full linux
> dual boot.
>
> Hmm.
>
> > In a more general note, it seems that WSL2 will be able to replace
> windows specific build very effectively
> > in the very near future.
>
> For everything other than OpenGL graphics (and maybe the Cloud would also
> be problematic), yes, it seems so.
>
> Paul.
>



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

Re: [ccp4bb] Coot under WSL2

[Clement Degut]

Immediate update :

If you use VcXserv, you need to start it with all default option, and
generate a config file.
Then change in the file wgl to False.
Restart computer (yes)
And start VcXserv with the modified config.
If you simply deactivate native openGl via the gui nothing is working
anymore for some reason, it must change another option somewhere.

doing that the display of coot is perfectly fine, just a bit laggy, as i
guess it's full software render, but usable (with a beefy CPU).

Clément

On Thu, 19 Mar 2020 at 14:48, Clement Degut 
wrote:

> The performance are massively improved in WSL2 versus WSL1, so maybe it's
> why you had bad memory of it ?
> WSL2 is currently only available in insider fast ring, but will be for
> everyone with next major update (v2004) in few weeks.
>
> The main problem is, while you can use the right toolbar by simply
> detaching it, selecting residues is almost impossible as the pointer is
> still using the main UI as reference, but the 3D display  is away, so you
> need to click far from the residus you try to select.
>
> I am playing around with the xserver options, but am not sure what I am
> doing with that windows X server.
> I'll update if I find a solution.
>
> Clément
>
> On Thu, 19 Mar 2020 at 13:31, Paul Emsley 
> wrote:
>
>> On 19/03/2020 12:43, Clement Degut wrote:
>> >
>> > I have been recently trying to use ccp4 in windows subsystem for Linux
>> 2 (Ubuntu distribution), everything i
>> > tried run perfectly smoothly, including graphical interfaces.
>>
>> I am pleasantly surprised - when I tried that (some time ago) the
>> graphics performance as terrible. Perhaps
>> you have better hardware.
>>
>> > My only problem is with coot (latest build), the interface is working
>> fine, as well as the 3D graphical. The
>> > only problem is that the graphical window is offset from the rest of
>> the UI (see screenshot).
>> > So although it looks like everything would work smoothly it's not
>> really usable.
>>
>> What, would you say, is the biggest problem?
>>
>> > I guess it comes from the absence of GPU access under WSL2 and so no
>> proper display driver, but does anyone
>> > had this problem? (and managed to solve it).
>>
>> FWIW, I have not seen this problem before (and don't know how to fix it)
>>
>> It seems to me an X server problem (or configuration issue). Have you
>> tweaked or disabled the native OpenGL
>> options on the X server?
>>
>> > Would be very useful for people working from home without a full linux
>> dual boot.
>>
>> Hmm.
>>
>> > In a more general note, it seems that WSL2 will be able to replace
>> windows specific build very effectively
>> > in the very near future.
>>
>> For everything other than OpenGL graphics (and maybe the Cloud would also
>> be problematic), yes, it seems so.
>>
>> Paul.
>>
>



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

[ccp4bb] displyaing cryo-EM map in pymol

[George Lountos]

I wanted to ask if anyone has a good reference or tutorial link for displaying  
and working with cryo-EM map files in pymol.

Thanks,
George




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

Re: [ccp4bb] displyaing cryo-EM map in pymol

[Steve Chou]

Hi George,
The following commands use to work for me.
=
# on computer terminal
gzip ~/Desktop/pore_loops/myProt.mrc
# compress .mrc file into .gz file; it will generate a file named
myProt.mrc.gz

load  the pdb into PyMOL
# on PyMOL terminal
hide all
show cartoon
set cartoon_oval_length, 1
# make helices slimmer

load ~/Desktop/pore_loops/myProt.mrc.gz, vvv_map, format=ccp4
isomesh vvv_mesh, vvv_map, 0.1
color blue, vvv_mesh
=
Steve

On Thu, Mar 19, 2020 at 4:02 PM George Lountos  wrote:

> I wanted to ask if anyone has a good reference or tutorial link for
> displaying  and working with cryo-EM map files in pymol.
>
> Thanks,
> George
>
>
> --
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1
>


-- 
Steve Chou



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

[ccp4bb] Job Post - PostDoc position (CSSB, Hamburg, Germany) for the project ‘Structural and mechanistic studies of functional protein fibrils using correlative electron and light microscopy’

[Landau Meytal]

Dear CCP4 scientists,


This is a joint collaborative project between Israel and the University of 
Hamburg on the "mechanisms governing virulence and antibacterial properties of 
protein fibrils in infectious and aggregation disease"s.

The research methods include single particle cryoEM reconstruction, correlative 
light and electron microscopy, artificial membrane systems (supported lipid 
bilayers, liposomes), and live cell microscopy to study toxicity mechanisms of 
fibrils on bacteria and mammalian cells and within microbial biofilms.

Project leaders are Associate Prof. Meytal Landau, Technion-Israel Institute of 
Technology; Prof. Dr. rer. nat. Kay Grünewald, Heinrich-Pette-Institute, 
Leibniz Institute of Experimental Virology and Universität Hamburg, Fakultät 
für Mathematik, Informatik und Naturwissenschaften (MIN) and Centre for 
Structural Systems Biology (CSSB), DESY, Hamburg; Dr. Carolin Seuring, Head of 
the CryoEM Multi-User Facility, CSSB; Dr. Roland Thünauer, Head of the Advanced 
Light and Fluorescence Microscopy Facility, CSSB.

The research will be conducted at the CSSB on the Bahrenfeld Research Campus, 
Hamburg, Germany, also the home of the German synchrotron DESY, with occasional 
visits to Israel. CSSB harbors state-of-the-art multi-user facilities for 
protein production and characterization, high-throughput crystallization, 
advanced light microscopy and electron microscopy.

For more information: https://www.cssb-hamburg.de/facilities/index_eng.html

Requirements:

  *   PhD in the field of structural biology, biophysics, or related fields

  *   Fluent in English (both written and oral)
  *   Experience in protein science and structural biology
  *   Desirable experience in cryo-EM and/or light microscopy
  *   The ability to work independently and as part of a team
  *   A high level of motivation
  *   Self-driven and organized

The position commences as soon as possible and is a fixed-term contract for a 
period of 2 years.

For applications (by 1/4/2020) and further information, please contact Prof. 
Meytal Landau (mlan...@technion.ac.il) and Prof. 
Dr. rer. nat. Kay Grünewald 
(kay.gruenew...@cssb-hamburg.de).

For further information on the Multi-User CryoEM and ALFM -facilities, please 
visit our website.

Best,

Meytal on behalf of the collaborative team




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1