Hi, Looking at your save state test_suite/shared_data/dispersion/bug_21665.bz2 and your printout, it looks like the specific_analyses.relax_disp. disp_data.loop_exp_frq_offset_point_time() function is not operating correctly. Therefore the bug must be in one of the loop_() functions! You could, if you feel adventurous, create unit tests for the loop_() functions by simply loading the bug_21665.bz2 saved state, executing the loop_() function, and then checking that the values and indices are what you would expect. This is like in your message, but instead of using a print() call, you use an assertEqual() call and have all the values you would expect stored in a list at the start of the unit test function. These unit tests would really speed up the debugging by being much closer to the bug. I'll create the unit test directories and empty files required now - you can chose to use them if you wish.
As for storing the data, that is not the problem. The base is the experiment ID string which is used as a key in dictionaries for each piece of information (experiment type, spectrometer frequency, offset, dispersion point, and time point are all separate dictionaries). The data storage is quite basic as you can see in the saved state. This has the advantage of being stable over the long term for save state compatibility between different relax versions. The data storage in the relax data store is designed to be as simple as possible. The problem is the construction of loops over this data - this is the complicated part. Obviously, for the data you have, one or more of these loops are not functioning correctly. You are getting much closer to the real issue! If you have unit tests for each of the loop_() functions, then once these pass you should eliminate all issues with this data combination. Regards, Edward On 19 February 2014 12:29, Troels Emtekær Linnet <[email protected]> wrote: > Hi Edward. > > If i use the function: > > for exp_type, frq, offset, point, time, ei, mi, oi, di, ti in > specific_analyses.relax_disp.disp_data.loop_exp_frq_offset_point_time(return_indices=True): > print exp_type, frq, offset, point, time, ei, mi, oi, di, ti > > Where i guess that: > ei = experiment index > mi = frequency index > oi = offset index > di = dispersion index > ti = time index > > SQ CPMG 499862140.0 0.0 50.0 0.04 0 0 0 0 0 > SQ CPMG 499862140.0 0.0 50.0 0.06 0 0 0 0 1 > SQ CPMG 499862140.0 0.0 100.0 0.04 0 0 0 1 0 > SQ CPMG 499862140.0 0.0 100.0 0.06 0 0 0 1 1 > SQ CPMG 499862140.0 0.0 150.0 0.04 0 0 0 2 0 > SQ CPMG 499862140.0 0.0 150.0 0.06 0 0 0 2 1 > SQ CPMG 499862140.0 0.0 200.0 0.04 0 0 0 3 0 > SQ CPMG 499862140.0 0.0 200.0 0.06 0 0 0 3 1 > SQ CPMG 499862140.0 0.0 250.0 0.04 0 0 0 4 0 > SQ CPMG 499862140.0 0.0 250.0 0.06 0 0 0 4 1 > SQ CPMG 499862140.0 0.0 300.0 0.04 0 0 0 5 0 > SQ CPMG 499862140.0 0.0 300.0 0.06 0 0 0 5 1 > SQ CPMG 499862140.0 0.0 350.0 0.04 0 0 0 6 0 > SQ CPMG 499862140.0 0.0 350.0 0.06 0 0 0 6 1 > SQ CPMG 499862140.0 0.0 400.0 0.04 0 0 0 7 0 > SQ CPMG 499862140.0 0.0 400.0 0.06 0 0 0 7 1 > SQ CPMG 499862140.0 0.0 450.0 0.04 0 0 0 8 0 > SQ CPMG 499862140.0 0.0 450.0 0.06 0 0 0 8 1 > SQ CPMG 499862140.0 0.0 500.0 0.04 0 0 0 9 0 > SQ CPMG 499862140.0 0.0 500.0 0.06 0 0 0 9 1 > SQ CPMG 499862140.0 0.0 550.0 0.04 0 0 0 10 0 > SQ CPMG 499862140.0 0.0 550.0 0.06 0 0 0 10 1 > SQ CPMG 499862140.0 0.0 600.0 0.04 0 0 0 11 0 > SQ CPMG 499862140.0 0.0 600.0 0.06 0 0 0 11 1 > SQ CPMG 499862140.0 0.0 650.0 0.04 0 0 0 12 0 > SQ CPMG 499862140.0 0.0 650.0 0.06 0 0 0 12 1 > SQ CPMG 499862140.0 0.0 700.0 0.04 0 0 0 13 0 > SQ CPMG 499862140.0 0.0 700.0 0.06 0 0 0 13 1 > SQ CPMG 499862140.0 0.0 800.0 0.04 0 0 0 14 0 > SQ CPMG 499862140.0 0.0 800.0 0.06 0 0 0 14 1 > SQ CPMG 499862140.0 0.0 900.0 0.04 0 0 0 15 0 > SQ CPMG 499862140.0 0.0 900.0 0.06 0 0 0 15 1 > SQ CPMG 499862140.0 0.0 1000.0 0.04 0 0 0 16 0 > SQ CPMG 499862140.0 0.0 1000.0 0.06 0 0 0 16 1 > SQ CPMG 599890858.7 0.0 33.333 0.04 0 1 0 0 0 > SQ CPMG 599890858.7 0.0 33.333 0.06 0 1 0 0 1 > SQ CPMG 599890858.7 0.0 66.667 0.04 0 1 0 1 0 > SQ CPMG 599890858.7 0.0 66.667 0.06 0 1 0 1 1 > SQ CPMG 599890858.7 0.0 100.0 0.04 0 1 0 2 0 > SQ CPMG 599890858.7 0.0 100.0 0.06 0 1 0 2 1 > SQ CPMG 599890858.7 0.0 133.333 0.04 0 1 0 3 0 > SQ CPMG 599890858.7 0.0 133.333 0.06 0 1 0 3 1 > SQ CPMG 599890858.7 0.0 166.667 0.04 0 1 0 4 0 > SQ CPMG 599890858.7 0.0 166.667 0.06 0 1 0 4 1 > SQ CPMG 599890858.7 0.0 200.0 0.04 0 1 0 5 0 > SQ CPMG 599890858.7 0.0 200.0 0.06 0 1 0 5 1 > SQ CPMG 599890858.7 0.0 233.333 0.04 0 1 0 6 0 > SQ CPMG 599890858.7 0.0 233.333 0.06 0 1 0 6 1 > SQ CPMG 599890858.7 0.0 266.667 0.04 0 1 0 7 0 > SQ CPMG 599890858.7 0.0 266.667 0.06 0 1 0 7 1 > SQ CPMG 599890858.7 0.0 300.0 0.04 0 1 0 8 0 > SQ CPMG 599890858.7 0.0 300.0 0.06 0 1 0 8 1 > SQ CPMG 599890858.7 0.0 333.333 0.04 0 1 0 9 0 > SQ CPMG 599890858.7 0.0 333.333 0.06 0 1 0 9 1 > SQ CPMG 599890858.7 0.0 366.667 0.04 0 1 0 10 0 > SQ CPMG 599890858.7 0.0 366.667 0.06 0 1 0 10 1 > SQ CPMG 599890858.7 0.0 400.0 0.04 0 1 0 11 0 > SQ CPMG 599890858.7 0.0 400.0 0.06 0 1 0 11 1 > SQ CPMG 599890858.7 0.0 466.667 0.04 0 1 0 12 0 > SQ CPMG 599890858.7 0.0 466.667 0.06 0 1 0 12 1 > SQ CPMG 599890858.7 0.0 533.333 0.04 0 1 0 13 0 > SQ CPMG 599890858.7 0.0 533.333 0.06 0 1 0 13 1 > SQ CPMG 599890858.7 0.0 666.667 0.04 0 1 0 14 0 > SQ CPMG 599890858.7 0.0 666.667 0.06 0 1 0 14 1 > SQ CPMG 599890858.7 0.0 866.667 0.04 0 1 0 15 0 > SQ CPMG 599890858.7 0.0 866.667 0.06 0 1 0 15 1 > SQ CPMG 599890858.7 0.0 1000.0 0.04 0 1 0 16 0 > SQ CPMG 599890858.7 0.0 1000.0 0.06 0 1 0 16 1 > > Then we see that the dispersion point of 50.0 exist for time 0.04 and 0.06. > That is not true, this point only exists for time point 0.04 > > Maybe I am reading your suggestion a little wrong. > > I am a little un-sure if relax stores the data in the wrong way, and > this should be handled. > Or if the problem is the "checks" before the calculation. > > And then I guess that there are some looping functions which collect > the data, before > doing a calculation. > > > > > > > 2014-02-18 15:50 GMT+01:00 Edward d'Auvergne <[email protected]>: >> Ah, that's the problem. You'll need to replace this with one of the >> relaxation dispersion loops, have a look at the loop_*() functions in >> specific_analyses.relax_disp.disp_data. You could for example use the >> loop_exp_frq_offset_point() and then count the time points for each of >> these iterations. This is only one of many alternatives. >> >> If you feel more adventurous, you could make the count_relax_times() >> function dependent on ei, mi, oi, and di (see the module docstring for >> a description of this: >> http://www.nmr-relax.com/api/3.1/specific_analyses.relax_disp.disp_data-module.html). >> You would then have to delete the use of loop_time() and do something >> as in the insignificance() function. >> >> There are other ways too. The key is to loop over each experiment >> collection of {ei, mi, oi, di} and check the times for each. If there >> is only one time for the experiment block, then this is the 2-point >> calculation. If there is more, then exponential curve fitting is >> needed. If one block of {ei, mi, oi, di} is 2-point and another is >> exponential, then relax will fail. But this type of data combination >> is very unlikely to ever be encountered and we can deal with that when >> a user complains ;) >> >> Anyway, have a look at specific_analyses.relax_disp.disp_data for >> inspiration, as all the code you need to write can be copied from >> various functions of that module. The logic you need is repeated in >> about 10 functions there. I hope this helps. >> >> Regards, >> >> Edward >> >> >> >> >> On 18 February 2014 15:35, Troels Emtekær Linnet <[email protected]> >> wrote: >>> Hi Edward. >>> >>> Do you have a feeling for how much modification of relax is needed? >>> >>> There is a dependency check, that the length of cdp.relax_time_list is >>> not larger than 1. >>> >>> Is it within range to allow more than one delay time? >>> >>> Best >>> Troels >>> >>> 2014-02-18 14:21 GMT+01:00 Edward d'Auvergne <[email protected]>: >>>> Hi Troels, >>>> >>>> I would then recommend you create 2 tests, as it looks to me like the >>>> fundamental issues are the same - that the detection of 2-point verses >>>> full exponential curves is not functioning correctly. The two tests, >>>> once the bug is fixed, will the make sure that both these errors will >>>> never again be seen by relax users! >>>> >>>> Cheers, >>>> >>>> Edward >>>> >>>> >>>> >>>> On 18 February 2014 12:40, Troels Emtekær Linnet <[email protected]> >>>> wrote: >>>>> If the systemtest is modified to: >>>>> >>>>> self.interpreter.calc(verbosity=1) >>>>> >>>>> The error is: >>>>> >>>>> Traceback (most recent call last): >>>>> File >>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/test_suite/system_tests/relax_disp.py", >>>>> line 278, in test_bug_21665_cpmg_two_fields_two_delaytimes_fail >>>>> self.interpreter.calc(verbosity=1) >>>>> File >>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/prompt/uf_objects.py", >>>>> line 221, in __call__ >>>>> self._backend(*new_args, **uf_kargs) >>>>> File >>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/pipe_control/minimise.py", >>>>> line 86, in calc >>>>> calculate(verbosity=verbosity) >>>>> File >>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/specific_analyses/relax_disp/api.py", >>>>> line 717, in calculate >>>>> self._calculate_r2eff() >>>>> File >>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/specific_analyses/relax_disp/api.py", >>>>> line 182, in _calculate_r2eff >>>>> check_exp_type_fixed_time() >>>>> File >>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/specific_analyses/relax_disp/checks.py", >>>>> line 112, in check_exp_type_fixed_time >>>>> raise RelaxError("The experiment '%s' is not of the fixed >>>>> relaxation time period data type." % exp_type) >>>>> RelaxError: RelaxError: The experiment 'SQ CPMG' is not of the fixed >>>>> relaxation time period data type. >>>>> >>>>> >>>>> 2014-02-18 12:24 GMT+01:00 Troels Emtekær Linnet <[email protected]>: >>>>>> Hi Edward. >>>>>> >>>>>> The systemtest is now implemented. >>>>>> I think this bug is related to: >>>>>> bug #21344 (https://gna.org/bugs/?21344): Handling of in sparse >>>>>> acquired R1rho dataset with missing combinations of time and spin-lock >>>>>> field strengths. >>>>>> >>>>>> The output of the systemtest gives: >>>>>> ------------ >>>>>> Fitting to spin :2@N, frequency 499862140.0 and dispersion point 50.0 >>>>>> --------------------------------------------------------------------- >>>>>> >>>>>> Unconstrained grid search size: 9 (constraints may decrease this size). >>>>>> >>>>>> Traceback (most recent call last): >>>>>> File >>>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/test_suite/system_tests/relax_disp.py", >>>>>> line 281, in test_bug_21665_cpmg_two_fields_two_delaytimes_fail >>>>>> relax_disp.Relax_disp(pipe_name="compare_128_FT_R2eff", >>>>>> pipe_bundle="cpmg_disp_sod1d90a", results_dir=self.tmpdir, >>>>>> models=['R2eff'], grid_inc=3, mc_sim_num=5, modsel='AIC', >>>>>> pre_run_dir=None, insignificance=1.0, numeric_only=False, >>>>>> mc_sim_all_models=False, eliminate=True) >>>>>> File >>>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/auto_analyses/relax_disp.py", >>>>>> line 116, in __init__ >>>>>> self.run() >>>>>> File >>>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/auto_analyses/relax_disp.py", >>>>>> line 451, in run >>>>>> self.optimise(model=model) >>>>>> File >>>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/auto_analyses/relax_disp.py", >>>>>> line 350, in optimise >>>>>> self.interpreter.grid_search(inc=self.grid_inc) >>>>>> File >>>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/prompt/uf_objects.py", >>>>>> line 221, in __call__ >>>>>> self._backend(*new_args, **uf_kargs) >>>>>> File >>>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/pipe_control/minimise.py", >>>>>> line 152, in grid_search >>>>>> grid_search(lower=lower, upper=upper, inc=inc, >>>>>> constraints=constraints, verbosity=verbosity) >>>>>> File >>>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/specific_analyses/relax_disp/api.py", >>>>>> line 1093, in grid_search >>>>>> self.minimise(min_algor='grid', lower=lower, upper=upper, inc=inc, >>>>>> constraints=constraints, verbosity=verbosity, sim_index=sim_index) >>>>>> File >>>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/specific_analyses/relax_disp/api.py", >>>>>> line 1142, in minimise >>>>>> self._minimise_r2eff(min_algor=min_algor, min_options=min_options, >>>>>> func_tol=func_tol, grad_tol=grad_tol, max_iterations=max_iterations, >>>>>> constraints=constraints, scaling=scaling, verbosity=verbosity, >>>>>> sim_index=sim_index, lower=lower, upper=upper, inc=inc) >>>>>> File >>>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/specific_analyses/relax_disp/api.py", >>>>>> line 382, in _minimise_r2eff >>>>>> values.append(average_intensity(spin=spin, exp_type=exp_type, >>>>>> frq=frq, offset=offset, point=point, time=time, sim_index=sim_index)) >>>>>> File >>>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/specific_analyses/relax_disp/disp_data.py", >>>>>> line 114, in average_intensity >>>>>> int_keys = find_intensity_keys(exp_type=exp_type, frq=frq, >>>>>> offset=offset, point=point, time=time) >>>>>> File >>>>>> "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/specific_analyses/relax_disp/disp_data.py", >>>>>> line 347, in find_intensity_keys >>>>>> raise RelaxError("No intensity data could be found corresponding >>>>>> to the spectrometer frequency of %s MHz, dispersion point of %s and >>>>>> relaxation time of %s s." % (frq*1e-6, point, time)) >>>>>> RelaxError: RelaxError: No intensity data could be found corresponding >>>>>> to the spectrometer frequency of 499.86214 MHz, dispersion point of >>>>>> 50.0 and relaxation time of 0.06 s. >>>>>> >>>>>> ----------------------------- >>>>>> >>>>>> This is true. >>>>>> For the relaxation time of 0.06 s, there is no dispersion point of 50.0. >>>>>> >>>>>> Best >>>>>> Troels >>>>>> >>>>>> 2014-02-18 12:19 GMT+01:00 <[email protected]>: >>>>>>> Author: tlinnet >>>>>>> Date: Tue Feb 18 12:19:41 2014 >>>>>>> New Revision: 22198 >>>>>>> >>>>>>> URL: http://svn.gna.org/viewcvs/relax?rev=22198&view=rev >>>>>>> Log: >>>>>>> Added systemtest to catch bug: relax -s >>>>>>> Relax_disp.test_bug_21665_cpmg_two_fields_two_delaytimes_fail >>>>>>> >>>>>>> Regarding bug #21665, (https://gna.org/bugs/?21665) - Running a CPMG >>>>>>> analysis with two fields at two delay times. >>>>>>> >>>>>>> Modified: >>>>>>> trunk/test_suite/system_tests/relax_disp.py >>>>>>> >>>>>>> Modified: trunk/test_suite/system_tests/relax_disp.py >>>>>>> URL: >>>>>>> http://svn.gna.org/viewcvs/relax/trunk/test_suite/system_tests/relax_disp.py?rev=22198&r1=22197&r2=22198&view=diff >>>>>>> ============================================================================== >>>>>>> --- trunk/test_suite/system_tests/relax_disp.py (original) >>>>>>> +++ trunk/test_suite/system_tests/relax_disp.py Tue Feb 18 12:19:41 2014 >>>>>>> @@ -264,6 +264,21 @@ >>>>>>> relax_disp.Relax_disp.opt_max_iterations = 1000 >>>>>>> relax_disp.Relax_disp(pipe_name="origin - relax_disp (Thu Jan >>>>>>> 2 13:46:44 2014)", pipe_bundle="relax_disp (Thu Jan 2 13:46:44 2014)", >>>>>>> results_dir=self.tmpdir, models=['R2eff', 'No Rex', 'CR72', 'NS CPMG >>>>>>> 2-site expanded'], grid_inc=3, mc_sim_num=5, modsel='AIC', >>>>>>> pre_run_dir=None, insignificance=1.0, numeric_only=False, >>>>>>> mc_sim_all_models=False, eliminate=True) >>>>>>> >>>>>>> + >>>>>>> + def test_bug_21665_cpmg_two_fields_two_delaytimes_fail(self): >>>>>>> + """U{Bug #21665<https://gna.org/bugs/?21665>} catch, the >>>>>>> failure due to a a CPMG analysis recorded at two fields at two delay >>>>>>> times.""" >>>>>>> + >>>>>>> + # Clear the data store. >>>>>>> + self.interpreter.reset() >>>>>>> + >>>>>>> + # Load the state. >>>>>>> + state = status.install_path + >>>>>>> sep+'test_suite'+sep+'shared_data'+sep+'dispersion'+sep+'bug_21665.bz2' >>>>>>> + self.interpreter.state.load(state, force=True) >>>>>>> + >>>>>>> + # Execute the auto-analysis (fast). >>>>>>> + relax_disp.Relax_disp.opt_func_tol = 1e-5 >>>>>>> + relax_disp.Relax_disp.opt_max_iterations = 1000 >>>>>>> + relax_disp.Relax_disp(pipe_name="compare_128_FT_R2eff", >>>>>>> pipe_bundle="cpmg_disp_sod1d90a", results_dir=self.tmpdir, >>>>>>> models=['R2eff'], grid_inc=3, mc_sim_num=5, modsel='AIC', >>>>>>> pre_run_dir=None, insignificance=1.0, numeric_only=False, >>>>>>> mc_sim_all_models=False, eliminate=True) >>>>>>> >>>>>>> >>>>>>> def test_curve_type_cpmg_fixed_time(self): >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> relax (http://www.nmr-relax.com) >>>>>>> >>>>>>> This is the relax-commits mailing list >>>>>>> [email protected] >>>>>>> >>>>>>> To unsubscribe from this list, get a password >>>>>>> reminder, or change your subscription options, >>>>>>> visit the list information page at >>>>>>> https://mail.gna.org/listinfo/relax-commits >>>>> >>>>> _______________________________________________ >>>>> relax (http://www.nmr-relax.com) >>>>> >>>>> This is the relax-devel mailing list >>>>> [email protected] >>>>> >>>>> To unsubscribe from this list, get a password >>>>> reminder, or change your subscription options, >>>>> visit the list information page at >>>>> https://mail.gna.org/listinfo/relax-devel _______________________________________________ relax (http://www.nmr-relax.com) This is the relax-devel mailing list [email protected] To unsubscribe from this list, get a password reminder, or change your subscription options, visit the list information page at https://mail.gna.org/listinfo/relax-devel
