Dear Gerard, As an industry user of an EIGER 16M at the SLS beamline X06SA and using autoPROC as the processing component of our pipeline, I would like to fully support what you have written. There was a serious gap in software capabilities when the EIGER was mounted and luckily Clemens and Claus jumped in to fill this gap by writing a super-fast (full) hdf5-to-cbf converter to enable us to process data with autoPROC. They also set up native hdf5-support for autoPROC very quickly after that. Thanks a lot for that. We would not have been able to keep the standards with regard to speed and quality of results without the help of Global Phasing.
Best wishes, Dirk --- Dr. Dirk Reinert Boehringer Ingelheim Pharma GmbH & Co. KG Lead Identification and Opt. Support Tel.: +49 (7351) 54-97892 Fax: +49 (7351) 83-97892 mailto:dirk.rein...@boehringer-ingelheim.com Boehringer Ingelheim Pharma GmbH & Co. KG, Sitz: Ingelheim am Rhein; Registergericht Mainz: HR A 22206; Komplementär Boehringer Ingelheim Deutschland GmbH; Geschäftsführung: Stefan Rinn (Vorsitzender), Ursula Fuggis-Hahn, Ralf Gorniak, Dr. Douglas Khoo, Andreas Krüger; Vorsitzender des Aufsichtsrates: Dr. Joachim Hasenmaier; Sitz: Ingelheim am Rhein; Registergericht Mainz: HR B 23260 Diese E-Mail ist vertraulich zu behandeln. Sie kann besonderem rechtlichem Schutz unterliegen. Wenn Sie nicht der richtige Adressat sind, senden Sie bitte diese E-Mail an den Absender zurück, löschen die eingegangene E-Mail und geben den Inhalt der E-Mail nicht weiter. Jegliche unbefugte Bearbeitung, Nutzung, Vervielfältigung oder Verbreitung ist verboten. / This e-mail is confidential and may also be legally privileged. If you are not the intended recipient please reply to sender, delete the e-mail and do not disclose its contents to any person. Any unauthorized review, use, disclosure, copying or distribution is strictly prohibited. -----Ursprüngliche Nachricht----- Von: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] Im Auftrag von Gerard Bricogne Gesendet: Freitag, 11. März 2016 09:25 An: CCP4BB@JISCMAIL.AC.UK Betreff: Re: [ccp4bb] Eigers, and local CBF formats Dear Graeme, It is good news that you, Herb, the NeXuS people and Dectris have been working hard towards the lofty goals you describe. However, much weeping and gnashing of teeth would have been spared to many people if that effort had come to fruition by the time images started gushing out of the first Eiger detectors on beamlines, some nine months ago or more. If what you are in the process of designing and writing had been available then, we would have been very happy not to have to dive into these very technical issues. However, we had to write such a converter because (1) XDS doesn't support HDF5 natively and requires an extractor utility, (2) the initial implementation of this converter by Dectris (H5ToXds) had some problems at the time (that are now fixed as far as we know), (3) we also wanted to support OsX and not be limited to 64-bit Linux and (4) several of our users required a fast converter that would write fully populated and correct mini-cbf headers. We took great care in writing those fully populated and correct mini-cbf headers. We haven't seen mini-cbf/CBF files from every one of the converters that are out there, but some of the files we see from other converters are much less complete and are clearly intended to be used only as an intermediate for a particular program. So even if the primary use of our converter is also to provide intermediate files (hidden from the user) for processing with autoPROC/XDS, at least these files are intended to (and _should_) be fully standardised and allow processing with any other package that supports mini-cbf/CBF files. Regarding autoPROC itself, we are not proposing that users convert HDF5 files into mini-cbf/CBF files before running it - the documentation is very clear about that: users should give autoPROC the HDF5 data directly. You may deplore the "profusion" of these various types of files and say that "this way madness lies", but we feel the major madness in this whole story may well have been that of creating a fabulous piece of hardware, having not made sufficient provision for software to be ready to deliver what the users were expecting from it. In such a situation, it seems to us that any group of people who pull up their sleeves and try to help fill that software vacuum, with the right sense of urgency and in a way that tries to also provide tools that people other than just themselves can use, are contributing something valuable that may legitimately expect to attract if not appreciation, then at least some respect, rather than being dismissed as having only contributed to general pollution of the field. After that, of course, things can take their time to settle, and we all look forward to the Experts coming up with the Mother of All Standards along with an implementation thereof that will inspire future generations. Nine months or so ago, however, priorities looked very different, and what seems to you unsatisfactory today was seen as very satisfactory then, precisely because it did fulfil an immediate (and urgent) need for some people who were keen to process their Eiger images "at home". Finally, we are not sure what exactly the word "software" means in your last sentence. You talk about a data processing package, but it can't designate autoPROC, as this is universally available, free of charge, to academics - showing that open-source development funded by public money is not the only model of scientific software development that can put good tools in the hands of the scientific community for free. We feel that a key ingredient in creating "a vibrant ecosystem" of methods developers is to acknowledge the diversity that can exist between approches that can all bear fruit in their own way. With best wishes, Clemens, Claus and Gerard. -- On Thu, Mar 10, 2016 at 07:06:49AM +0000, graeme.win...@diamond.ac.uk wrote: > Dear Gerard, > > An ideal we have been working towards within DIALS/xia2 and also with Herbert > Bernstein, the NeXuS people & Dectris (and many others) is to (i) define and > use the full HDF5 data formats on MX beamlines and (ii) develop the data > processing software to natively support these detectors. This work is > ongoing, however is hampered as you indicate below by the profusion of > different data formats which exist for the HDF5 container itself. One issue > which has become apparent however is *in addition* to this population of HDF5 > formats there is a further collection of nearly-mini-cbf files being created, > some of which are presumably coming from your software, some from the > software Harry mentioned and I would guess a further population from locally > developed tools. > > It is clear that this way madness lies, and we are headed in that direction > with great enthusiasm! > > Our original motivation in asking for these data was to allow us to provide > support in DIALS and xia2 for these files where possible, including the > native HDF5. A secondary benefit of this, beyond the software itself being > available to the user community (Google will give anyone interested the > necessary pointers) is that the code itself which interprets the data is > available, under a free open source license, allowing (i) peer review of the > code itself to ensure the understanding is correct (ii) the opportunity for > such peer to actually contribute to getting fixes in place and {most > importantly} (iii) an implicit definition of the format for future > generations of developers in the absence of any kind of explicit format > definitions. > > I for one welcome multiple options for all data processing challenges, as > this makes for a vibrant ecosystem and ensures that the end user community > are most likely to get the best from their data. This has been a particular > strength of MX over the past decades. I therefore feel that “you can process > these data with (fill in any one package here)” is unsatisfactory, even if it > may fulfil an immediate need. I feel this all the more strongly when the > software in question is not universally available. > > Best wishes Graeme > > > On 9 Mar 2016, at 16:52, Gerard Bricogne <g...@globalphasing.com> wrote: > > > > Dear Graeme and Eiger users, > > > > We have been grasping this nettle since last Summer, and it is a > > particularly stinging one. At least, crystallographers who have a > > need to process Eiger datasets right now can do this with autoPROC > > with a fair chance of success. The remaining problems that are > > turning up at the moment have to do not just with incomplete > > information in locally produced miniCBF files but with > > inconsistencies between information of differing provenance about the same > > items. > > > > We know of a number of beamlines and users that make use of the > > HDF5 converter now included in autoPROC for the task of generating > > miniCBF files. If you install the latest (snapshot) version of > > autoPROC by going to > > > > http://www.globalphasing.com/autoproc/ > > > > you will have access not only to the native Eiger/HDF5 support in > > autoPROC, but also to the latest version of our converter utility. > > See > > > > http://www.globalphasing.com/autoproc/ReleaseNotes/ReleaseNotes-auto > > PROC_snapshot_20160304.txt > > http://www.globalphasing.com/autoproc/wiki/index.cgi?DataProcessingH > > df5 > > > > We hope that the resulting CBF files (with full mini-cbf header) > > would be fully compatible with other processing packages, viewers or > > tools - if they can't read HDF5 datasets directly. > > > > Please let us know if there are inconsistencies or issues. This > > invitation extends to all users of this HDF5 capability in autoPROC. > > > > > > With best wishes, > > > > Clemens, Claus and Gerard. > > > > -- > > On Tue, Mar 08, 2016 at 07:57:57PM +0000, Graeme Winter wrote: > >> Hello World, > >> > >> A few times over the last week we at the xia2 helpdesk have been asked > >> about support for Eiger detectors, and particularly local dialects of CBF > >> file which are created on Eiger powered beamlines. Often we are able to > >> get a single image which is helpful but from there it is hard to prove > >> everything is working right. > >> > >> Please could beamline scientists who have an Eiger and use local software > >> to make this into miniCBF files get in touch, ideally with an example data > >> set, so we can test the support in xia2 and DIALS and make corrections > >> where necessary. > >> > >> Thanks & best wishes Graeme > >> -- > >> This e-mail and any attachments may contain confidential, copyright and or > >> privileged material, and are for the use of the intended addressee only. > >> If you are not the intended addressee or an authorised recipient of the > >> addressee please notify us of receipt by returning the e-mail and do not > >> use, copy, retain, distribute or disclose the information in or attached > >> to the e-mail. > >> Any opinions expressed within this e-mail are those of the individual and > >> not necessarily of Diamond Light Source Ltd. > >> Diamond Light Source Ltd. cannot guarantee that this e-mail or any > >> attachments are free from viruses and we cannot accept liability for any > >> damage which you may sustain as a result of software viruses which may be > >> transmitted in or with the message. > >> Diamond Light Source Limited (company no. 4375679). Registered in > >> England and Wales with its registered office at Diamond House, > >> Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 > >> 0DE, United Kingdom
