Storing all the images *is* expensive but it can be done - the JCSG do this and make available a good chunk of their raw diffraction data. The cost is, however, in preparing this to make the data useful for the person who downloads it.
If we are going to store and publish the raw experimental measurements (e.g. the images) which I think would be spectacular, we will also need to define a minimum amount of metadata which should be supplied with this to allow a reasonable chance of reproduction of the results. This is clearly not trivial, but there is probably enough information in the harvest and log files from e.g. CCP4, HKL2000, Phenix to allow this. The real problem will be in getting people to dig out that tape / dvd with the images on, prepare the required metadata and "deposit" this information somewhere. Actually storing it is a smaller challenge, though this is a long way from being trivial. On an aside - firewire disks are indeed a very cheap way of storing the data. There is a good reason why they are much cheaper than the equivalent RAID array. They fail. Ever lost 500GB of data in one go? Ouch. ;o) Just MHO. Cheers, Graeme -----Original Message----- From: CCP4 bulletin board [mailto:[EMAIL PROTECTED] On Behalf Of Phil Evans Sent: 16 August 2007 15:13 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] The importance of USING our validation tools What do you count as raw data? Rawest are the images - everything beyond that is modellling - but archiving images is _expensive_! Unmerged intensities are probably more manageable Phil On 16 Aug 2007, at 15:05, Ashley Buckle wrote: > Dear Randy > > These are very valid points, and I'm so glad you've taken the > important step of initiating this. For now I'd like to respond to one > of them, as it concerns something I and colleagues in Australia are > doing: >> >> The more information that is available, the easier it will be to >> detect fabrication (because it is harder to make up more information >> convincingly). For instance, if the diffraction data are deposited, >> we can check for consistency with the known properties of real >> macromolecular crystals, e.g. that they contain disordered solvent >> and not vacuum. As Tassos Perrakis has discovered, there are >> characteristic ways in which the standard deviations depend on the >> intensities and the resolution. If unmerged data are deposited, there >> will probably be evidence of radiation damage, weak effects from >> intrinsic anomalous scatterers, etc. Raw images are probably even >> harder to simulate convincingly. > > After the recent Science retractions we realised that its about time > raw data was made available. So, we have set about creating the > necessary IT and software to do this for our diffraction data, and are > encouraging Australian colleagues to do the same. We are about a week > away from launching a web-accessible repository for our recently > published (eg deposited in PDB) data, and this should coincide with an > upcoming publication describing a new structure from our labs. The aim > is that publication occurs simultaneously with release in PDB as well > as raw diffraction data on our website. > We hope to house as much of our data as possible, as well as data from > other Australian labs, but obviously the potential dataset will be > huge, so we are trying to develop, and make available freely to the > community, software tools that allow others to easily setup their own > repositories. After brief discussion with PDB the plan is that PDB > include links from coordinates/SF's to the raw data using a simple > handle that can be incorporated into a URL. We would hope that we can > convince the journals that raw data must be made available at the time > of publication, in the same way as coordinates and structure factors. > Of course, we realise that there will be many hurdles along the way > but we are convinced that simply making the raw data available ASAP is > a 'good thing'. > > We are happy to share more details of our IT plans with the CCP4BB, > such that they can be improved, and look forward to hearing feedback > > cheers
