Hi all, The major changes described in Richard's previous mail have now been merged into the ARTS master branch. As mentioned, we don't expect everything to work perfectly right out of the box. Please give the new version a try with your own controlfiles and provide us feedback.
There is a branch (before-newlinerecord) available in the GitHub repository that points to the ARTS version just before this merge. You can check it out if you need to revert your local ARTS version back to the old system. You can also use the "Files changed" tab in this github diff view[1] to see what changes have been made and how the existing controlfiles have been adapted to the new system: [1] https://github.com/atmtools/arts/compare/before-newlinerecord...master <https://github.com/atmtools/arts/compare/before-newlinerecord...master> One amendment to Richard's previous mail concerning iyEmissionStandardParallel: iyEmissionStandard has been replaced with iyEmissionStandardParallel. Therefore, there is no need to change anything related to that in your controlfiles. The old version of iyEmissionStandard is currently still available under the name iyEmissionStandardSequential, but it is considered deprecated and should be removed after further testing. Cheers, Oliver > On 9. Dec 2019, at 09:40, Richard Larsson <ric.lars...@gmail.com> wrote: > > Dear ARTS developers, > > The infrastructure for ARTS line-by-line data will change significantly with > an upcoming update to the development version of ARTS. Your simulations will > be broken before you adapt to the new infrastructure. I will use this email > to explain what updates are required to continue your simulations as before. > I will first justify the necessity of the update. To follow this in more > detail, please see the discussion between various people involved in the > update on the github page. The discussion is named "newlinerecord #70". It > might be particularly useful to check how different *.arts files has had to > be updated. > > Background > > In the past few years the line-by-line data has grown massively. The reason > for this is the necessity to include quantum numbers to do some of the more > advanced calculations, and the change from only allowing pure Voigt > calculations in air, to allow any species mixture to make up the broadening > bath, to allow line mixing computations, and to allow other beyond-Voigt line > shapes. > > The main problem with the old system of keeping all data on a line-by-line > basis was that a lot of this data was shared between several lines. In > total, we stored close to 1 kB of data per absorption line that could > basically be shared between hundreds or thousands of other lines. This was > about half of the absorption line memory footprint. Another problem was that > each line could have their own reference temperature, their own mass, their > own pressure broadening scheme, their own cutoff frequencies, their own line > normalization, and so on. Thus every individual line must have many of their > parameters computed from scratch even though it was likely that much of the > information of the previous absorption line could just be copied. > > The new system intends to keep all the metadata about how the computations > are performed separate from the pure line-by-line data. This will create > more robust absorption line catalogs, since the meta data about the > computations are stored with the original data itself. > > Controlfiles > > The problem for the controlfiles are however quite large. There are several > workspace variables that have been removed: abs_lineshape, > abs_species_per_band, lm_p_lim, and zeeman_linerecord_precalc. These are now > dealt with by the new absorption line format itself. For lm_p_lim, this > means that the entire band has to have a singular pressure limit. For > abs_lineshape, this means that the entire band has to share the same line > normalization, line cutoff, and line shape calculator. There are methods to > set these globally on a loaded catalog, either for a specific > ArrayOfSpeciesTag or for the entire line catalog. The others two just became > superfluous during the changes being made to the line catalog. > > There has been one new workspace variables added: lbl_checked. This variable > has to be set to run any of the line-by-line calculations, i.e., *Zeeman and > *AddLines. There is an lbl_checkedCalc() method available for this purpose. > > The most major change is in how the absorption lines are stored. The old > abs_lines and abs_lines_per_species still exists. They are now arrays of > AbsorptionLines rather than arrays of LineRecord. The reading routines for > this has changed. There are several Read* functions that output the > abs_lines variable. They should be used to read the data. > > Lastly, the NLTE variables has had to become a class of their own instead of > compromised of several matching variables. This is because the matching > variables used to be kept in the line-by-line data before but cannot do that > now. The reading routines for the atmospheric data now has to have several > new inputs. See the artscomponent/nlte folder for details on how to do this. > > Several methods have also been changed or removed. Particularly, if they > existed only to set some of the workspace variables above, then they are, > perhaps obviously, gone. But one notable change that might affect some users > is that the abs_xsec_per_speciesAddLines2 no longer exist as its purpose is > now the standard in abs_xsec_per_speciesAddLines. > > Example change > > As an example of the changes you need to do, from the TestClearSky2.arts > test-file, which used to read: > > abs_lineshapeDefine( shape="Voigt_Kuntz6", forefactor="VVH", cutoff=750e9 ) > # in general.arts > ReadXML( abs_lines, "abs_lines.xml" ) > > now reads: > > ReadARTSCAT( abs_lines=abs_lines, filename="abs_lines.xml" ) > abs_linesSetCutoff(abs_lines, "ByLine", 750e9) > abs_linesSetNormalization(abs_lines, "VVH") > ... # after most of the other parts have been defined > lbl_checkedCalc > > Note that a major difference here is that "Voigt_Kuntz6" no longer exist as > an option for these calculations. The method is simply incompatible with > reliable internal Jacobian calculations because it does not produce the > required phase. Note that storing this catalog after the two abs_linesSet*() > calls will store the cutoff information and the normalization information so > that if ReadXML() is used on the new catalog, there will be no need to call > the *Set*() functions again. > > The exact line catalog reader you need to use depends on your use case. All > previous versions of ARTSCAT can be read, and all possible catalog we could > read in the past we can still read today. > > Speed > > Note that some of you might experience changes in the speed of some > calculations without also adopting other parts of the code. The parallel > paths through the code has had to change. We aim to address this in the > future but the code as it will be committed will not be fully > speed-compatible to some of the old use-cases. In particular, it is advised > for now to switch from iyEmissionStandard to iyEmissionStandardParallel. > This switch is already implemented in agenda.arts. > > Generally, however, the new code either outperforms or matches the old code. > > The end? > > I am sure there will be many hiccups and troubles as this major branch is > merged, but I hope they can be addressed without much problem as we go > forward. > > With hope, > //Richard > _______________________________________________ > arts_dev.mi mailing list > arts_dev.mi@lists.uni-hamburg.de > https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_dev.mi
_______________________________________________ arts_dev.mi mailing list arts_dev.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_dev.mi
