[arts-users] Changing base parameter for abs_lines
Dear Richard,
Thank you for your quick and very detailed explanations.
Following your suggestion, I had another look at the Perrin spectroscopic files
(NewFormat) and for O3-666, the localquanta="J" parameter in the
AbsorptionLines xml-tag is already defined as such.
So I am not sure if I should just append the fake quantum number to the lines
of interest, which produced the following error when I tried to read the
modified file version with the ReadXML method:
"Error parsing rational number"
or if maybe they should be defined in a specific column within the file ?
Thanks also for your advise regarding the Jacobian methods. I'll have a look at
these as soon as the quantum identifier or "catalog identity" works (no ways to
use these method without right ?) but this is probably what I am looking for my
sensitivity analysis.
Thanks for you help,
Eric
De : arts_users.mi de la part de
arts_users.mi-requ...@lists.uni-hamburg.de
Envoyé : jeudi 17 février 2022 12:00
À : arts_users.mi@lists.uni-hamburg.de
Objet : arts_users.mi Digest, Vol 67, Issue 10
Send arts_users.mi mailing list submissions to
arts_users.mi@lists.uni-hamburg.de
To subscribe or unsubscribe via the World Wide Web, visit
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
or, via email, send a message with subject or body 'help' to
arts_users.mi-requ...@lists.uni-hamburg.de
You can reach the person managing the list at
arts_users.mi-ow...@lists.uni-hamburg.de
When replying, please edit your Subject line so it is more specific
than "Re: Contents of arts_users.mi digest..."
Today's Topics:
1. Re: Changing base parameter for abs_lines (Richard Larsson)
--
Message: 1
Date: Wed, 16 Feb 2022 15:43:40 +0100
From: Richard Larsson
To: eric.sauvag...@unibe.ch
Cc: arts users mi
Subject: Re: [arts-users] Changing base parameter for abs_lines
Message-ID:
Content-Type: text/plain; charset="utf-8"
Dear Eric,
The QI is not available for the data you are looking at.
It stands for QuantumIdentifier, which is the class in Arts that deals with
quantum numbers and gas species for the purpose of identifying a unique
absorption line. In 2.4, you define this as something like "O3-666 TR UP J
1 LO J 2", for the gas species O3-666 with upper quantum number J=1 and
lower quantum number J=2. I am not sure in 2.4 if you can simply define a
QuantumIdentifier in pyarts or if you are required to load it from a file.
You can see example files in the spectroscopy/zeeman/*.qid.xml files for
how an energy level is defined with this format (the EN tag makes the UP
and LO tags for different levels unnecessary).
Since the Perrin data does not provide this id-data, the lines are not
unique by Arts standards (or any standard really). Thus, you cannot use
these methods to change any line parameters directly. I'm afraid you have
to use more manual methods. I will make a suggestion below.
You could define 'fake' quantum numbers for the lines that you are
interested in. If you set localquanta="J" in the AbsorptionLines xml-tag,
and then append "0 0" for the first absorption line, "1 1" for the second,
and so on for all the absorption lines. Now the lines that you are
interested in can be found using "O3-666 TR UP J i LO J i", where i would
be the absorption line index. This way you could perform the calculations
that you are after and use the built-in methods to change the base
parameters for the lines you identify as important.
Note though that if you go with this solutions and your interest is just to
compute the sensitivities, there are two Jacobian methods available for
that purpose - jacobianAddShapeCatalogParameter
and jacobianAddBasicCatalogParameter - that already analytically computes
what the partial derivative wrt the added catalog parameters will be.
Using them is the same as using any other type of Jacobian in Arts, so you
should be able to compute the sensitivities in a somewhat straight-forward
manner. (The catalog_identity for these are the QI of the other
methods.). You can add as many unique parameters (for multiple absorption
lines) you want and the output should just be OK.
With hope,
//Richard
Den tors 10 feb. 2022 kl 07:40 skrev :
> Dear ARTS community,
>
>
> In order to perform some sensitivity analysis on ozone retrievals, I would
> like to artificially modify some absorption lines parameters (e.g. line
> strength and frequencies) on the ozone line.
>
>
> From my understanding, this would be the goal of either ones the following
> workspace methods:
>
>
> abs_linesChangeBaseParameterForMatchingLines()
>
>
> abs_lines_per_speciesChangeBaseParameterForSpecies()
>
>
> which both require among a QI ("quantum identifier to match the line")
> parameter.
>
>
> If this is the case, I am not sure about what exactly is this QI and where
> I can find it (or set it) for
[arts-users] Changing base parameter for abs_lines
Dear ARTS community,
In order to perform some sensitivity analysis on ozone retrievals, I would like
to artificially modify some absorption lines parameters (e.g. line strength and
frequencies) on the ozone line.
>From my understanding, this would be the goal of either ones the following
>workspace methods:
abs_linesChangeBaseParameterForMatchingLines()
abs_lines_per_speciesChangeBaseParameterForSpecies()
which both require among a QI ("quantum identifier to match the line")
parameter.
If this is the case, I am not sure about what exactly is this QI and where I
can find it (or set it) for a specific absorption line ?
I am using ARTS 2.4 (through pyarts) and am defining the spectroscopy by
reading the Perrin_newformat_speciessplit/o3-666.xml.gz located in the original
ARTS-xml-data folder (2.4) and then uning the
abs_lines_per_speciesCreateFromLines() method.
Thanks for your help.
Best,
Eric
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
[arts-users] Error during inversion_agenda with pyarts
Dear ARTS developers,
I am updating my retrievals at the moment to switch to pyarts 2.4.0 and make it
work with the stable ARTS v2.4.x.
It seems to work because I get the same results as "previously" but I still get
an TypeError message each time the inversion_iterate_agenda is run, i.e. at
each OEM iteration.
By previously I mean 2 cases:
1. Using the Typhon ARTS interface and ARTS 2.4.0
2. Using pyarts 2.5.0 built with ARTS 2.4.x but a few commits behind
(actually b1ca4e0ff8756e416c38096cd1b9b678cb041c30). On this one, I believe was
building the last ARTS master branch but still runs with ARTS2.4.x
Note that the error does not prevent the OEM to continue and to converge to a
value which seems right to me (and similar to previous runs without this
logging error) but still I think it would be nice to understand why it's
appearing and if this is something to worry about.
I copied the error OEM function call, the inversion_agenda definition and the
full error message (a bit nasty sorry). The last part (in red) is maybe already
enough to understand what is wrong though.
I hope this is not too confusing and I thank you in advance for your time.
Best,
Eric
##
My OEM call:
ws.OEM(method='gn', max_iter=10, stop_dx=20, lm_ga_settings=lm_ga_settings,
display_progress=1)
##
My inversion agenda:
@arts_agenda
def gromora_inversion_agenda(ws):
"""Custom inversion iterate agenda to ignore bad partition
functions."""
ws.Ignore(ws.inversion_iteration_counter)
ws.xClip(ijq=0, limit_low=0.001, limit_high=0.2)
# Map x to ARTS' variables
ws.x2artsAtmAndSurf()
ws.x2artsSensor()
# To be safe, rerun some checkss
ws.atmfields_checkedCalc(negative_vmr_ok=True)
ws.atmgeom_checkedCalc()
# Calculate yf and Jacobian matching x
ws.yCalc() # ws.yCalc(y=ws.yf)
# Add baseline term
#ws.VectorAddElementwise(ws.yf, ws.y, ws.y_baseline)
ws.VectorAddVector(ws.yf, ws.y, ws.y_baseline)
# This method takes cares of some "fixes" that are needed to get
the Jacobian
# right for iterative solutions. No need to call this WSM for
linear inversions.
ws.jacobianAdjustAndTransform()
##
The error message:
--- Logging error ---
Traceback (most recent call last):
File "/opt/ARTS/arts2.4.x/build/python/pyarts/workspace/workspace.py", line
135, in callback
eval(compile(m , "", 'exec'), context)
TypeError: required field "type_ignores" missing from Module
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/opt/anaconda/miniconda3/envs/GROMORA_retrievals/lib/python3.8/logging/__init__.py",
line 1085, in emit
msg = self.format(record)
File
"/opt/anaconda/miniconda3/envs/GROMORA_retrievals/lib/python3.8/logging/__init__.py",
line 929, in format
return fmt.format(record)
File
"/opt/anaconda/miniconda3/envs/GROMORA_retrievals/lib/python3.8/logging/__init__.py",
line 668, in format
record.message = record.getMessage()
File
"/opt/anaconda/miniconda3/envs/GROMORA_retrievals/lib/python3.8/logging/__init__.py",
line 373, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File
"/opt/anaconda/miniconda3/envs/GROMORA_retrievals/lib/python3.8/runpy.py", line
194, in _run_module_as_main
return _run_code(code, main_globals, None,
File
"/opt/anaconda/miniconda3/envs/GROMORA_retrievals/lib/python3.8/runpy.py", line
87, in _run_code
exec(code, run_globals)
File
"/home/es19m597/.vscode/extensions/ms-python.python-2021.12.1559732655/pythonFiles/lib/python/debugpy/__main__.py",
line 45, in
cli.main()
File
"/home/es19m597/.vscode/extensions/ms-python.python-2021.12.1559732655/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py",
line 444, in main
run()
File
"/home/es19m597/.vscode/extensions/ms-python.python-2021.12.1559732655/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py",
line 285, in run_file
runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
File
"/opt/anaconda/miniconda3/envs/GROMORA_retrievals/lib/python3.8/runpy.py", line
265, in run_path
return _run_module_code(code, init_globals, run_name,
File
"/opt/anaconda/miniconda3/envs/GROMORA_retrievals/lib/python3.8/runpy.py", line
97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File
"/opt/anaconda/miniconda3/envs/GROMORA_retrievals/lib/python3.8/runpy.py", line
87, in _run_code
exec(code, run_globals)
File
"/home/es19m597/Documents/GROMORA/GROMORA-harmo/scripts/retrieval/retrieve.py",
line 203, in
ac, retrieval_param, sensor_out =
Re: [arts-users] Perrin spectroscopy
Dear Stefan, Thanks for the quick answer, that's perfect ! Best regards, Eric De : Stefan Buehler Envoyé : vendredi 24 septembre 2021 09:58:50 À : Sauvageat, Eric (IAP) Cc : arts_users.mi@lists.uni-hamburg.de Objet : Re: [arts-users] Perrin spectroscopy Dear Eric, there is a paper by Agnès about them, here attached. Best wishes, Stefan On 24 Sep 2021, at 9:49, eric.sauvag...@iap.unibe.ch wrote: > Dear ARTS community, > > > As I am interested with the O3 line at 142GHZ which has somehow a weird > frequency center value in pure HITRAN, I am using since the beginning your > so-called "Perrin_newformat_speciessplit" in ARTS data which is a mix between > HITRAN and Perrin spectroscopic parameters (with O3 coming from Perrin with > correct freq. center). > However, I could not really find any source for the "Perrin" spectroscopic > parameters so I would like to ask if you could provide a reference to it or > some more information about where these parameters comes from ? > Also may I ask which version of HITRAN was used to do this mixed > spectroscopic catalogue ? > > Thanks a lot, > Eric > ___ > arts_users.mi mailing list > arts_users.mi@lists.uni-hamburg.de > https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
[arts-users] Perrin spectroscopy
Dear ARTS community, As I am interested with the O3 line at 142GHZ which has somehow a weird frequency center value in pure HITRAN, I am using since the beginning your so-called "Perrin_newformat_speciessplit" in ARTS data which is a mix between HITRAN and Perrin spectroscopic parameters (with O3 coming from Perrin with correct freq. center). However, I could not really find any source for the "Perrin" spectroscopic parameters so I would like to ask if you could provide a reference to it or some more information about where these parameters comes from ? Also may I ask which version of HITRAN was used to do this mixed spectroscopic catalogue ? Thanks a lot, Eric ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] Tropospheric continuum retrieval in ARTS
Dear Patrick, Thanks a lot for the help and feedback, this is really appreciated. For the second order term of the baseline, I believe one of the reason for it lies in our sideband suppression method. I'll now try to add that effect in the sensor_response variable and see if this impacts the baseline accordingly. Thanks, Eric De : Patrick Eriksson Envoyé : mardi 22 juin 2021 20:39:56 À : Sauvageat, Eric (IAP); arts_users...@mailman.rrz.uni-hamburg.de Cc : rita.edit.kaj...@ltu.se Objet : Re: [arts-users] Tropospheric continuum retrieval in ARTS Eric, > (1998)). The values of the poly_order=0 are now quite small (mostly > lower than 0.1K), however, I still only get positive values. Is that > something that could be expected from the way the function > "retrievalAddPolyfit" works maybe ? No, there is nothing that promotes positive values. But you now get so small values that nothing to worry about. You get very consistent values for poly_order 1 and 2, despite that the tropospheric opacity varies quite a bit. That should be an indication on that receiver is stable and the baseline+tropospheric part of the retrievals is working well. > Would you have any tips on how to implement the retrieval of a single > scaling parameter for a full water vapor profile ? I believe this is > more or less what the CONTABS_DO parameter was implementing according to > the first Qpack documentation ? This you do by setting the retrieval grid to have a single point (e.g. 500 hPa) and set the corresponding covariance matrix to have size 1x1. A suitable value for it is maybe 1 (meaning a 100% uncertainty at all altitudes, fully correlated). Bye, Patrick ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] Tropospheric continuum retrieval in ARTS
Hi, Thanks, I'll see if I can reduce that value by lowering the apriori cov for the polyfit order 0 then. So the "single continuum fit" was really just setting a single altitude and retrieving a singe H2O VMR value at this altitude whereas the profile case was the same retrievals (species, apriori cov) but with an apriori profile instead of a single value. I'll check the rel unit then, I was not aware this could be used to retrieved a single scaling value on a profile. Best regards, Eric De : Patrick Eriksson Envoyé : vendredi 11 juin 2021 10:23:58 À : Sauvageat, Eric (IAP); arts_users...@mailman.rrz.uni-hamburg.de Objet : Re: [arts-users] Tropospheric continuum retrieval in ARTS Eric, It seems indeed that poly_order=0 is involved. I think you want this value to be small. Especially as it seems to only be positive. This behaviour would be OK if you thought that you have a calibration error that is random, but just one-sided. Sounds not reasonable. I think more realistic is to trust the calibration and let the tropospheric part take care of this part. poly_order=0 and the tropospheric correction have a very similar impact on the "baseline". The resulting tropospheric attenuation depends to some extent on the retrieved humidity profile, but that should be secondary. But you can of course take a look at how the tropospheric humidity profile look in both cases and compare. By the way, how do you do the single column case? I suggest to use the rel unit. So you scale the profile with a single relative value. Using VMR for this is less realistic (here you would change the full profile with the same VMR value, that is not realistic). Regards, Patrick On 2021-06-11 10:02, eric.sauvag...@iap.unibe.ch wrote: > Dear Patrick, > > > Thanks for you answer. > > > I did some further investigations on the value of the polyfit and as > expected the value of the constant terms is indeed quite different for > both options and very similar for 1st and 2nd degree terms (I appended > some basic representative plots to give an idea). > > > More specifically, it is smaller in the case of the "single continuum > fit" which is the one giving a +10% ozone profile. It would mean > that tropospheric attenuation is considered to be larger in the case of > the single continuum fit compared to the other option. I think this is > then consistent with the 10% higher ozone values obtained in this case, > would you agree ? > > > I remember having tried different constraint on the constant term but > only making it larger so I should redo some test reducing it more. I > believe one way to check if the tropospheric attenuation is correct is > to compare the opacity value computed from ARTS to one computed manually > on the spectra. > > I'll check that as well but thanks for your inputs already. > > > Best regards, > > Eric > > > > > > > *De :* Patrick Eriksson > *Envoyé :* mercredi 9 juin 2021 23:39:33 > *À :* Sauvageat, Eric (IAP); arts_users...@mailman.rrz.uni-hamburg.de > *Objet :* Re: [arts-users] Tropospheric continuum retrieval in ARTS > Eric, > > I don't know about a fixed setup for dealing with the troposphere in > observations of this type. As your observations at max gives one piece > of information for the troposphere, I would say that the single > retrieval point setup makes most sense. In any case it is plausible option. > > I assume that you fit the measurements in both cases. > > If I get it right, your main worry is that you get a 10% difference in > the ozone profile between the options. This should likely originate in > that the retrievals give you a troposphere having a 10% difference in > transmission. > > My suggestion is then to look at the polyfit part. Is the fitted polyfit > the same between the options? My guess is that it differs. And that > gives room for retrieving a different tropospheric transmission. > > And it could be reasons to anyhow consider the polyfit part. I assume > you have a good calibration and the uncertainty in the overall > "baseline" level is due to the troposphere. Or expressed differently, > you want to fit the overall baseline level by changing H2O in the > troposphere, not by the polyfit. Or more exactly you want the first > polyfit coefficient to be small, the polyfit should just take care of > the "wiggling". To achieve this you should set the a priori uncertainty > for the first (0-order) coefficient to be very small, to effectively > enforce a low measurement response for the coefficient. With this, the > retrieval will have to fully adjust the overall baseline level by the > H2O profile, independently on the grid and a priori uncertainty you use > for H2O. > > Bye, > > Patrick > > > > > > On 2021-06-09 16:21, eric.sauvag...@iap.unibe.ch wrote: >> Dear ARTS community, >> >> >> I am doing stratospheric O3 retrieval with a ground-based radiometer >> (f=142GHz) and
Re: [arts-users] Tropospheric continuum retrieval in ARTS
Dear Mathias, Nice, I am happy to see that other people are working on this topic as well. In terms of apriori, I am indeed doing the same kind of forcing on the ozone profile to avoid some high ozone VMR retrieved at low altitudes, problem that I got before using such a forcing. May I ask what you are using as apriori profile and covariance matrix for the H2O profile ? I tested ECMWF and Fascod as apriori for the H2O and it did not produce significant changes, the thing that matters the most in that view seems to be the apriori cov value for H2O setup with constant std deviation around 6e-4 or 6 ppm (I do all my retrievals in vmr units, both for O3 and H2O). Note that this value was mostly a guess but works (enable convergence...) for all my retrievals. This might change soon though depending on the current discussion.. Regarding the polyfit value, I am also not providing any specific apriori value, only a covariance matrix for each of the terms. Best regards, Eric De : Mathias Milz Envoyé : jeudi 10 juin 2021 08:05:53 À : Sauvageat, Eric (IAP); arts_users.mi@lists.uni-hamburg.de Cc : Rita Edit Kajtar; Uwe Raffalski Objet : Re: Tropospheric continuum retrieval in ARTS Dear Eric, We are working with a similar setup (Just at another frequency) for the radiometers in Kiruna. For water vapour we use a rather loose constraint (large S_x/S_a value) to allow the fit of the profile for the H2O background (co-fit). Of course, H2O can be unrealistic as we just use it to fit the continuum/baseline For O3 we use an altitude dependent constraint with a rather strong constraint below the tropopause (where we know the values are “negligible”) and above the sensitive area where we force the retrieval to the a priori knowledge. So we allow reasonable fitting only for the altitudes where the Jacobians show decent sensitivity. Note: We have good results with “rel” jacobians. “vmr”/absolute Jacobians caused problems. We also co-fit a baseline using “polyfit” of the lowest degree. However here we did not yet find a solution that we can start with a preselected value (e.g. an instrument-dependend baseline) so the a priori value is by default 0 and the a standard constrain constraint might cause difficulties with large large baselines due to high humidity or clouds with precipitation. Maybe this helps. Best regards, Mathias From: arts_users.mi-boun...@lists.uni-hamburg.de on behalf of eric.sauvag...@iap.unibe.ch Date: Wednesday, 9 June 2021 at 16:22 To: arts_users.mi@lists.uni-hamburg.de Subject: [arts-users] Tropospheric continuum retrieval in ARTS Dear ARTS community, I am doing stratospheric O3 retrieval with a ground-based radiometer (f=142GHz) and am trying to deal with the absorption contribution of the troposphere directly in the OEM implemented in ARTS (avoiding tropospheric correction prior to the retrievals). Up to now, I took inspiration from "qpack2_demo2.m" which suggests (if I understood it correctly) to implement a "H2O-PWR98, H2O" retrieval (main contributor of tropospheric opacity at these frequencies) on a lower atmosphere retrieval grid. This results in a water vapor profile retrieved together with my main ozone retrievals. Of course this profile has no good measurement response as my ozone radiometer is not designed to retrieve any H2O profile, but it seems that it provides the "right amount of opacity" needed to explain my spectrum. In addition, note that I am also performing a polyfit retrieval of degree 2 which is also fitting a constant term on my spectra which also probably contributes somehow to fit the global continuum absorption. I found out recently, that such a continuum retrieval was implemented in QPack1 (activated with "Q.CONTABS_DO") and from my understanding, it does not seem to retrieve any H2O profiles but only single values for the continuum (which somehow makes more sense to me). So I did try to provide a single grid retrieval point and single value for H2O cov matrix and it seems to work equally good as the retrieval including a full H2O profile (in the sense of convergence, correlation between both time series, ...) but it has a constant +10% VMR offset on my whole ozone profiles and I have no clue why. Also, I have made different tests to check the impact of the selected species (continuum vs full absorption model defined with or without H2O) but it did only produce slight changes in the results. As well, the height of the H2O grid or its altitude resolution does not seem to have significant impact on the retrievals. Sorry for this long email but I am really puzzled in what is the best way to deal with continuum absorption in ARTS and what I might be doing wrong. Therefore, any kind of feedback or help regarding this would be much appreciated. If needed, I can also provide examples plots of MR, AVK or profiles (not sure how it works for
[arts-users] Tropospheric continuum retrieval in ARTS
Dear ARTS community, I am doing stratospheric O3 retrieval with a ground-based radiometer (f=142GHz) and am trying to deal with the absorption contribution of the troposphere directly in the OEM implemented in ARTS (avoiding tropospheric correction prior to the retrievals). Up to now, I took inspiration from "qpack2_demo2.m" which suggests (if I understood it correctly) to implement a "H2O-PWR98, H2O" retrieval (main contributor of tropospheric opacity at these frequencies) on a lower atmosphere retrieval grid. This results in a water vapor profile retrieved together with my main ozone retrievals. Of course this profile has no good measurement response as my ozone radiometer is not designed to retrieve any H2O profile, but it seems that it provides the "right amount of opacity" needed to explain my spectrum. In addition, note that I am also performing a polyfit retrieval of degree 2 which is also fitting a constant term on my spectra which also probably contributes somehow to fit the global continuum absorption. I found out recently, that such a continuum retrieval was implemented in QPack1 (activated with "Q.CONTABS_DO") and from my understanding, it does not seem to retrieve any H2O profiles but only single values for the continuum (which somehow makes more sense to me). So I did try to provide a single grid retrieval point and single value for H2O cov matrix and it seems to work equally good as the retrieval including a full H2O profile (in the sense of convergence, correlation between both time series, ...) but it has a constant +10% VMR offset on my whole ozone profiles and I have no clue why. Also, I have made different tests to check the impact of the selected species (continuum vs full absorption model defined with or without H2O) but it did only produce slight changes in the results. As well, the height of the H2O grid or its altitude resolution does not seem to have significant impact on the retrievals. Sorry for this long email but I am really puzzled in what is the best way to deal with continuum absorption in ARTS and what I might be doing wrong. Therefore, any kind of feedback or help regarding this would be much appreciated. If needed, I can also provide examples plots of MR, AVK or profiles (not sure how it works for mailing list though). Many thanks in advance, Eric ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] ARTS OEM multiple inversions
Hi Patrick, Thanks for your quick answer. Simon's answer did clarify it and I believe his solution is quite close to what you suggest and what is done in Qpack. I'll try it out. Thanks, Eric De : Simon Pfreundschuh Envoyé : vendredi 4 juin 2021 14:55:24 À : Sauvageat, Eric (IAP); arts_users...@mailman.rrz.uni-hamburg.de Objet : Re: ARTS OEM multiple inversions Sorry, though the response would automatically go to the list. For completeness, here's my response from earlier. Just to clarify and confirm: > same sensors, same atmosphere, etc.. just different measurements vectors using > OEM from ARTS directly ? This is what ARTS currently does: It takes multiple observations and solves for a single atmospheric state. If I understand you correctly what you want to do is same sensors but different atmospheres (although they may be the same a priori). This kind of parallelism is not supported within ARTS so you'll have to do it sequentially. It's straight forward when you use the Python interface: After you have set up your calculation, you just call the OEM WSM multiple times in a loop. What you need to pay attention to is to reset the WSVs x, yf and jacobian before calling OEM a second time because those are both in- and outputs of the method. I not sure, however, how much you will gain from this computationally since this depends on your retrieval setup. Let me know if anything remains unclear. Kind regards, Simon From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de on behalf of eric.sauvag...@iap.unibe.ch Sent: Friday, June 4, 2021 12:03:12 PM To: arts_users...@mailman.rrz.uni-hamburg.de Subject: [arts-users] ARTS OEM multiple inversions Dear ARTS community, I am performing strato-mesospheric ozone retrievals with the OEM implemented directly in ARTS (using the python interface). As my sensor is not changing much with time, I am wondering if there is a way (and if this is worth in terms of computation speed) to reuse the sensor_response matrix (FFT channel response, unbinned) for different measurements which I think was quite straightforward with Qpack. I was trying to write multiple calibrated spectra within the y vector as well as multiple meas. error cov. (Se and inv) blocks before running the OEM which I thought might be the way to go but it seems to treat them as multiple measurements of the same ozone profiles instead. Could somebody confirmed if this is expected and if yes, is there another way to do this kind of parallel OEM retrievals: same sensors, same atmosphere, etc.. just different measurements vectors using OEM from ARTS directly ? Thanks for your help, Eric ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
[arts-users] ARTS OEM multiple inversions
Dear ARTS community, I am performing strato-mesospheric ozone retrievals with the OEM implemented directly in ARTS (using the python interface). As my sensor is not changing much with time, I am wondering if there is a way (and if this is worth in terms of computation speed) to reuse the sensor_response matrix (FFT channel response, unbinned) for different measurements which I think was quite straightforward with Qpack. I was trying to write multiple calibrated spectra within the y vector as well as multiple meas. error cov. (Se and inv) blocks before running the OEM which I thought might be the way to go but it seems to treat them as multiple measurements of the same ozone profiles instead. Could somebody confirmed if this is expected and if yes, is there another way to do this kind of parallel OEM retrievals: same sensors, same atmosphere, etc.. just different measurements vectors using OEM from ARTS directly ? Thanks for your help, Eric ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
Re: [arts-users] sensor_time definition through PyARTS
Dear Richard,
Thanks for the detailed explanation.
I used the following line as a workaround for now and it seems to work fine.
ws.timeNow()
ws.ArrayOfTimeSetConstant(ws.sensor_time, 1, ws.time)
No stress then but I am looking forward to your new implementation.
Thanks again and best regards,
Eric
De : Richard Larsson
Envoyé : mardi 16 février 2021 16:26:49
À : Patrick Eriksson
Cc : Sauvageat, Eric (IAP); arts_users...@mailman.rrz.uni-hamburg.de
Objet : Re: [arts-users] sensor_time definition through PyARTS
Hi,
This is my fault. Explanation if wanted: sensor_time is now a different type.
It now stores actual time stamps like "1970-01-01 01:00:01" (which is what "1"
means in time stamps if you are in CET).
I will fix it so you can set the time from a numpy array again in pyarts.
However, I don't understand the workspace interactions all that well. The
solution you will have to use before that is fixed will look like this:
pyarts.classes.from_workspace(ws.sensor_time).data = np.array([1])
or
pyarts.classes.from_workspace(ws.sensor_time).data = [1]
or
pyarts.classes.from_workspace(ws.sensor_time).data = ["2021-02-16 16:19:00"]
I am committing code for this shortly and it should be updated and merged soon.
Note that you have reading/writing routines for all ARTS variables via the
"pyarts.classes.from_workspace(x).savexml("filename.xml")" interface. In
addition about the new type for time, using
pyarts.classes.Time.TimeGrid(pyarts.classes.from_workspace(ws.sensor_time)),
you will be able to use matplotlib's time-series plotting features if you are
so inclined.
I hope we can find a way so that when there are no explicit workspace methods,
then the fall-back is to use the pyarts.classes interface but my python is not
that good to fix this myself.
Anyways, please give it a day or so for the code to do the above to be merged
before updating the development branch and you should have vector-interaction
back.
With hope,
//Richard
Den tis 16 feb. 2021 kl 15:05 skrev Patrick Eriksson
mailto:patrick.eriks...@chalmers.se>>:
Hi,
There is the same/similar problem on the Matlab side (as there is no
writing and reading of xml-files including the time group).
So a general hint for all. If you just want to set sensor_time to some
value, you can do this by these method calls
timeNow
ArrayOfTimeSetConstant( sensor_time, 1, time )
Bye,
Patrick
On 2021-02-16 14:53,
eric.sauvag...@iap.unibe.ch wrote:
> Dear ARTS community,
>
>
> In order to try a recent fix suggested and implemented (commit n.
> 5951a72fbef6d7ac9a3de8d2c503b58ef5af7d17, fromTyphon mailing list) on
> the development version of ARTS I came into another problem trying to
> implement my ground-based retrievals through PyARTS. In fact, I can't
> succeed anymore in setting the sensor_time variable through PyARTS,
> which is then required to perform the retrievals.
>
>
> I'm not sure if something got mixed maybe between the 2 ARTS version
> installed on my laptop and the 2 attached PyARTS versions (each in a
> separate conda env) or if this is really a bug from the development
> version.
>
>
> As a very short example, please find below a minimalist python script
> that reproduces the problem. It seems to work fine if I use ARTS2.4 and
> produces the following error with ARTS Dev version:
>
>
>
> ###
> import os
> import numpy as np
> import xarray as xr
> import matplotlib.pyplot as plt
>
> from pyarts.workspace import Workspace, arts_agenda
> from dotenv import load_dotenv
>
> #load_dotenv('/home/eric/Documents/PhD/ARTS/arts-examples/.env.t490-arts2.5')
> load_dotenv('/home/eric/Documents/PhD/ARTS/arts-examples/.env.t490-arts2.4')
> ARTS_DATA_PATH = os.environ['ARTS_DATA_PATH']
> ARTS_BUILD_PATH = os.environ['ARTS_BUILD_PATH']
> ARTS_INCLUDE_PATH = os.environ['ARTS_INCLUDE_PATH']
>
> def test_sensor():
> # Initializing Workspace object
> ws = Workspace(verbosity=0, agenda_verbosity=0)
> ws.execute_controlfile("general/general.arts")
> ws.execute_controlfile("general/agendas.arts")
> ws.execute_controlfile("general/continua.arts")
> ws.execute_controlfile("general/planet_earth.arts")
>
> ws.sensor_los = np.array([50])
> ws.sensor_pos = np.array([1e3])
> ws.sensor_time = np.array([1])
>
> print('sensor_time is :', ws.sensor_time.value)
>
> if __name__=="__main__":
> test_sensor()
>
> ###
>
>
>
> ___
> arts_users.mi mailing list
> arts_users.mi@lists.uni-hamburg.de
> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
>
___
arts_users.mi mailing list
Re: [arts-users] sensor_time definition through PyARTS
Dear Patrick,
Thanks a lot for the quick answer.
It seems indeed to define the sensor_time correctly in an ARTS file using your
2 lines.
On the PyARTS side, it seems that the defintion of ws.sensor_time is fine as
well but I can't access its value ("ws.sensor_time.value" fails, but not sure
if this is really something I need to do). It seems to me that this is related
to the lack of writing/reading routine for the time group as you mentionned
which might be corrected in the future so I believe this should be fine for now.
Thanks again,
Eric
De : Patrick Eriksson
Envoyé : mardi 16 février 2021 15:04:59
À : Sauvageat, Eric (IAP); arts_users...@mailman.rrz.uni-hamburg.de
Objet : Re: [arts-users] sensor_time definition through PyARTS
Hi,
There is the same/similar problem on the Matlab side (as there is no
writing and reading of xml-files including the time group).
So a general hint for all. If you just want to set sensor_time to some
value, you can do this by these method calls
timeNow
ArrayOfTimeSetConstant( sensor_time, 1, time )
Bye,
Patrick
On 2021-02-16 14:53, eric.sauvag...@iap.unibe.ch wrote:
> Dear ARTS community,
>
>
> In order to try a recent fix suggested and implemented (commit n.
> 5951a72fbef6d7ac9a3de8d2c503b58ef5af7d17, fromTyphon mailing list) on
> the development version of ARTS I came into another problem trying to
> implement my ground-based retrievals through PyARTS. In fact, I can't
> succeed anymore in setting the sensor_time variable through PyARTS,
> which is then required to perform the retrievals.
>
>
> I'm not sure if something got mixed maybe between the 2 ARTS version
> installed on my laptop and the 2 attached PyARTS versions (each in a
> separate conda env) or if this is really a bug from the development
> version.
>
>
> As a very short example, please find below a minimalist python script
> that reproduces the problem. It seems to work fine if I use ARTS2.4 and
> produces the following error with ARTS Dev version:
>
>
>
> ###
> import os
> import numpy as np
> import xarray as xr
> import matplotlib.pyplot as plt
>
> from pyarts.workspace import Workspace, arts_agenda
> from dotenv import load_dotenv
>
> #load_dotenv('/home/eric/Documents/PhD/ARTS/arts-examples/.env.t490-arts2.5')
> load_dotenv('/home/eric/Documents/PhD/ARTS/arts-examples/.env.t490-arts2.4')
> ARTS_DATA_PATH = os.environ['ARTS_DATA_PATH']
> ARTS_BUILD_PATH = os.environ['ARTS_BUILD_PATH']
> ARTS_INCLUDE_PATH = os.environ['ARTS_INCLUDE_PATH']
>
> def test_sensor():
> # Initializing Workspace object
> ws = Workspace(verbosity=0, agenda_verbosity=0)
> ws.execute_controlfile("general/general.arts")
> ws.execute_controlfile("general/agendas.arts")
> ws.execute_controlfile("general/continua.arts")
> ws.execute_controlfile("general/planet_earth.arts")
>
> ws.sensor_los = np.array([50])
> ws.sensor_pos = np.array([1e3])
> ws.sensor_time = np.array([1])
>
> print('sensor_time is :', ws.sensor_time.value)
>
> if __name__=="__main__":
> test_sensor()
>
> ###
>
>
>
> ___
> arts_users.mi mailing list
> arts_users.mi@lists.uni-hamburg.de
> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
>
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
[arts-users] sensor_time definition through PyARTS
Dear ARTS community,
In order to try a recent fix suggested and implemented (commit n.
5951a72fbef6d7ac9a3de8d2c503b58ef5af7d17, fromTyphon mailing list) on the
development version of ARTS I came into another problem trying to implement my
ground-based retrievals through PyARTS. In fact, I can't succeed anymore in
setting the sensor_time variable through PyARTS, which is then required to
perform the retrievals.
I'm not sure if something got mixed maybe between the 2 ARTS version installed
on my laptop and the 2 attached PyARTS versions (each in a separate conda env)
or if this is really a bug from the development version.
As a very short example, please find below a minimalist python script that
reproduces the problem. It seems to work fine if I use ARTS2.4 and produces the
following error with ARTS Dev version:
###
import os
import numpy as np
import xarray as xr
import matplotlib.pyplot as plt
from pyarts.workspace import Workspace, arts_agenda
from dotenv import load_dotenv
#load_dotenv('/home/eric/Documents/PhD/ARTS/arts-examples/.env.t490-arts2.5')
load_dotenv('/home/eric/Documents/PhD/ARTS/arts-examples/.env.t490-arts2.4')
ARTS_DATA_PATH = os.environ['ARTS_DATA_PATH']
ARTS_BUILD_PATH = os.environ['ARTS_BUILD_PATH']
ARTS_INCLUDE_PATH = os.environ['ARTS_INCLUDE_PATH']
def test_sensor():
# Initializing Workspace object
ws = Workspace(verbosity=0, agenda_verbosity=0)
ws.execute_controlfile("general/general.arts")
ws.execute_controlfile("general/agendas.arts")
ws.execute_controlfile("general/continua.arts")
ws.execute_controlfile("general/planet_earth.arts")
ws.sensor_los = np.array([50])
ws.sensor_pos = np.array([1e3])
ws.sensor_time = np.array([1])
print('sensor_time is :', ws.sensor_time.value)
if __name__=="__main__":
test_sensor()
###
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
[arts-users] ARTS-3
Hello, My name is Eric Sauvageat and I am a new PhD student in the MW Physics group at the Institute of Applied Physics of University Bern. My main task for the coming year it to set up a new harmonized retrieval for the two ground-based MW radiometers measuring Ozone profile in Switzerland. In this regards, the goal is to continue using ARTS as radiative transfer model and switch to ARTS-3 when it will be released. The current retrievals are using Matlab and Qpack for the interaction with ARTS and there is now the question arising if the new retrievals should be written on Matlab or switched to Python. In order to take a decision within the group, I would like to ask you the following questions regarding the future views for ARTS-3: 1. Will it be possible to continue using Qpack with the new release of ARTS and if yes, will it involve some major changes in the way of programming a new retrieval ? 2. What are (if existing) the long-term view of the ARTS developers regarding the use of Qpack with ARTS-3 ? Will it stay supported as the main way of interacting with ARTS or are there some inclinations towards some other ways (for instance using Typhon) ? I am looking forward to hearing from you and I thank you in advance for your answers. Best regards, Eric ___ arts_users.mi mailing list arts_users.mi@lists.uni-hamburg.de https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
