Re: [arts-users] Issues with size interpolation - ARTS SSD

[Jana Mendrok]

Hi Patrick et al,


on the same set of grids. That is, the idea is to use these functions
> first, before any size interpolation. This is at least indicated in the
> documentation of assp_interp_size.m.
>
>

> No time for me to dig into the Python code (and anyhow don't have the
> latest version at hand), and don't remember how closely Jana mimicked my
> Matlab interface.


Used to follow the Matlab implementation very closely. An assp_interp_za
exists. However, it's obviously nowhere applied (in the original Zenodo
published version; I did not check out newer ones).
The header of assp_interp_size states the need for common grids. Though I
agree, that's not very prominently, hence easily missed by a user and
definitely missed when not called directly by the user. Obviously,
assp_interp_size misses a proper check that this requirement is fulfilled...

I guess, back in the days all size instances of one habit had the same
number of angles, so it wasn't necessary to apply that (what speaks for
that is that an IconSnow.rssp file exists; and that had been created with
exactly the code that was published, ie without az-interpolation).

Isaac/Vasileios - get_assp hence needs to be extended with a call of
assp_interp_za, in between the assp_import_ssdb and assp.assp_interp_size
(not deep enough into it anymore to judge whether exact location
matters...).

In any case, Vasileios, thanks for caring for the code (and cleaning up
after me)!
Best,

-- 
Jana Mendrok, Ph.D.
Deutscher Wetterdienst
Offenbach am Main, Germany

+49 (0)69 8062 3139
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] Issues with size interpolation - ARTS SSD

[Isaac Moradi]

Hi Patrick,

I am not actually directly calling any assp functions. I use rttov.get_assp as 
shown below to get what I need and send the new grids for f, T, and dveq along 
with the habit ID (I have been playing with allow_extrap to just allow to fill 
the gap for now but I know it may not be a good idea to extrapolate). The 
interpolations inside rttov.py are done in the order that Patrick mentioned. I 
didn’t try to call individual functions for interpolation but in my code, I am 
using the same interpolation technique (PchipInterpolator) and of course as 
long as the variables have the correct size then it works just fine.

S,M,mD = rttov.get_assp(habID, orient, size_type='dveq',
size_grid = dveq_mw, s_allow_extrap=True,
t_grid=tgrid, t_allow_extrap=True,
f_grid=fgrid, f_allow_extrap=True)

Best,
Isaac



> On Jun 4, 2021, at 10:59 AM, Patrick Eriksson  
> wrote:
> 
> Hi,
> 
> Some quick input, for clarification.
> 
> At least for the Matlab interface there are functions for interpolating in 
> frequency, angular grid and temperature, to get the data of all sizes on the 
> same set of grids. That is, the idea is to use these functions first, before 
> any size interpolation. This is at least indicated in the documentation of 
> assp_interp_size.m.
> 
> No time for me to dig into the Python code (and anyhow don't have the latest 
> version at hand), and don't remember how closely Jana mimicked my Matlab 
> interface. But I should assume the same principle should be applied in 
> Python. Too bad if that not clear from the documentation.
> 
> Bye,
> 
> Patrick
> 
> 
> On 2021-06-04 16:22, Vasileios Barlakas wrote:
>> Hello Isaac,
>> I had a brief look at assp_interp_size. Indeed there are some bugs in this 
>> function that need to be fixed. I will allocate some time on this next week; 
>> sorry for any inconvenience caused. For the time being, I could give you 
>> some hints:
>> To use this function, all elements in S must have common frequency, 
>> temperature, and angular grid. Note here that, for some habits, e.g. 
>> IconSnow, the given sizes are not available in the same frequency and 
>> angular grids. This is one of the reasons why you get such an error message:
>> (31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6)
>> Note here that the function defines the dimensions of all the variables from 
>> the first element of the list, i.e., n_freq, n_temp, n_za, 1, 1, 1, 
>> n_phase_matrix_elements. For your case, the dimension at which python 
>> complains is the n_za (angular grid).
>> Following this rule, you could potentially fix any other issues. I hope this 
>> helps.
>> Best regards,
>> Vasileios
>> 
>> *From:* arts_users.mi-boun...@mailman.rrz.uni-hamburg.de 
>>  on behalf of Isaac Moradi 
>> 
>> *Sent:* Wednesday, June 2, 2021 4:36:50 PM
>> *To:* arts_users...@mailman.rrz.uni-hamburg.de
>> *Subject:* [arts-users] Issues with size interpolation - ARTS SSD
>> Hi all,
>> I am trying to do size interpolation using the assp.py included in the 
>> Python data interface but there seems to be multiple issues with the size 
>> interpolation function. There are some undefined variables that I fixed them 
>> but then end up with the following error.
>> I am trying this for IconSnow but the error should exist for any other 
>> habits.
>> Here is the message from Python:
>>   File "/data/users/imoradi/workspace/dda/DataInterfaces/Python/assp.py", 
>> line 1154, in assp_interp_size
>> p[i+1,...] = s.pha_mat_data
>> ValueError: could not broadcast input array from shape (31,5,721,1,1,1,6) 
>> into shape (31,5,91,1,1,1,6)
>> Thanks
>> Best,
>> Isaac
>> ___
>> arts_users.mi mailing list
>> arts_users.mi@lists.uni-hamburg.de
>> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
>> ___
>> arts_users.mi mailing list
>> arts_users.mi@lists.uni-hamburg.de
>> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
> ___
> arts_users.mi mailing list
> arts_users.mi@lists.uni-hamburg.de
> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] Issues with size interpolation - ARTS SSD

[Patrick Eriksson]

Hi,

Some quick input, for clarification.

At least for the Matlab interface there are functions for interpolating 
in frequency, angular grid and temperature, to get the data of all sizes 
on the same set of grids. That is, the idea is to use these functions 
first, before any size interpolation. This is at least indicated in the 
documentation of assp_interp_size.m.


No time for me to dig into the Python code (and anyhow don't have the 
latest version at hand), and don't remember how closely Jana mimicked my 
Matlab interface. But I should assume the same principle should be 
applied in Python. Too bad if that not clear from the documentation.


Bye,

Patrick


On 2021-06-04 16:22, Vasileios Barlakas wrote:

Hello Isaac,


I had a brief look at assp_interp_size. Indeed there are some bugs in 
this function that need to be fixed. I will allocate some time on this 
next week; sorry for any inconvenience caused. For the time being, I 
could give you some hints:



To use this function, all elements in S must have common frequency, 
temperature, and angular grid. Note here that, for some habits, e.g. 
IconSnow, the given sizes are not available in the same frequency and 
angular grids. This is one of the reasons why you get such an error message:



(31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6)


Note here that the function defines the dimensions of all the variables 
from the first element of the list, i.e., n_freq, n_temp, n_za, 1, 1, 1, 
n_phase_matrix_elements. For your case, the dimension at which python 
complains is the n_za (angular grid).



Following this rule, you could potentially fix any other issues. I hope 
this helps.



Best regards,

Vasileios



*From:* arts_users.mi-boun...@mailman.rrz.uni-hamburg.de 
 on behalf of Isaac 
Moradi 

*Sent:* Wednesday, June 2, 2021 4:36:50 PM
*To:* arts_users...@mailman.rrz.uni-hamburg.de
*Subject:* [arts-users] Issues with size interpolation - ARTS SSD
Hi all,

I am trying to do size interpolation using the assp.py included in the 
Python data interface but there seems to be multiple issues with the 
size interpolation function. There are some undefined variables that I 
fixed them but then end up with the following error.
I am trying this for IconSnow but the error should exist for any other 
habits.


Here is the message from Python:

   File 
"/data/users/imoradi/workspace/dda/DataInterfaces/Python/assp.py", line 
1154, in assp_interp_size

     p[i+1,...] = s.pha_mat_data
ValueError: could not broadcast input array from shape 
(31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6)



Thanks


Best,
Isaac
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi


___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] Issues with size interpolation - ARTS SSD

[Isaac Moradi]

HI Vasileios,

Thanks for the hints. Meanwhile, I spent sometime and wrote a piece of code to 
read the data in and interpolate them. I will they to see if can fix it 
according to what you suggested. In any case, once you have fixed the code, I 
will compare what I have developed with the main code. 

Thanks again.


Best,
Isaac





> On Jun 4, 2021, at 10:22 AM, Vasileios Barlakas 
>  wrote:
> 
> Hello Isaac,
> 
> I had a brief look at assp_interp_size. Indeed there are some bugs in this 
> function that need to be fixed. I will allocate some time on this next week; 
> sorry for any inconvenience caused. For the time being, I could give you some 
> hints:
> 
> To use this function, all elements in S must have common frequency, 
> temperature, and angular grid. Note here that, for some habits, e.g. 
> IconSnow, the given sizes are not available in the same frequency and angular 
> grids. This is one of the reasons why you get such an error message:
> 
> (31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6)
> 
> Note here that the function defines the dimensions of all the variables from 
> the first element of the list, i.e., n_freq, n_temp, n_za, 1, 1, 1, 
> n_phase_matrix_elements. For your case, the dimension at which python 
> complains is the n_za (angular grid).
> 
> Following this rule, you could potentially fix any other issues. I hope this 
> helps. 
> 
> Best regards,
> Vasileios
> 
> From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de 
>  on behalf of Isaac Moradi 
> 
> Sent: Wednesday, June 2, 2021 4:36:50 PM
> To: arts_users...@mailman.rrz.uni-hamburg.de
> Subject: [arts-users] Issues with size interpolation - ARTS SSD
>  
> Hi all,
> 
> I am trying to do size interpolation using the assp.py included in the Python 
> data interface but there seems to be multiple issues with the size 
> interpolation function. There are some undefined variables that I fixed them 
> but then end up with the following error.
> I am trying this for IconSnow but the error should exist for any other habits.
> 
> Here is the message from Python:
> 
>   File "/data/users/imoradi/workspace/dda/DataInterfaces/Python/assp.py", 
> line 1154, in assp_interp_size
> p[i+1,...] = s.pha_mat_data
> ValueError: could not broadcast input array from shape (31,5,721,1,1,1,6) 
> into shape (31,5,91,1,1,1,6)
> 
> 
> Thanks
> 
> 
> Best,
> Isaac
> ___
> arts_users.mi mailing list
> arts_users.mi@lists.uni-hamburg.de 
> https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi 
> 
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] Issues with size interpolation - ARTS SSD

[Vasileios Barlakas]

Hello Isaac,


I had a brief look at assp_interp_size. Indeed there are some bugs in this 
function that need to be fixed. I will allocate some time on this next week; 
sorry for any inconvenience caused. For the time being, I could give you some 
hints:


To use this function, all elements in S must have common frequency, 
temperature, and angular grid. Note here that, for some habits, e.g. IconSnow, 
the given sizes are not available in the same frequency and angular grids. This 
is one of the reasons why you get such an error message:


(31,5,721,1,1,1,6) into shape (31,5,91,1,1,1,6)


Note here that the function defines the dimensions of all the variables from 
the first element of the list, i.e., n_freq, n_temp, n_za, 1, 1, 1, 
n_phase_matrix_elements. For your case, the dimension at which python complains 
is the n_za (angular grid).


Following this rule, you could potentially fix any other issues. I hope this 
helps.


Best regards,

Vasileios



From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de 
 on behalf of Isaac Moradi 

Sent: Wednesday, June 2, 2021 4:36:50 PM
To: arts_users...@mailman.rrz.uni-hamburg.de
Subject: [arts-users] Issues with size interpolation - ARTS SSD

Hi all,

I am trying to do size interpolation using the assp.py included in the Python 
data interface but there seems to be multiple issues with the size 
interpolation function. There are some undefined variables that I fixed them 
but then end up with the following error.
I am trying this for IconSnow but the error should exist for any other habits.

Here is the message from Python:

  File "/data/users/imoradi/workspace/dda/DataInterfaces/Python/assp.py", line 
1154, in assp_interp_size
p[i+1,...] = s.pha_mat_data
ValueError: could not broadcast input array from shape (31,5,721,1,1,1,6) into 
shape (31,5,91,1,1,1,6)


Thanks


Best,
Isaac
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] ARTS OEM multiple inversions

[eric.sauvageat]

Hi Patrick,


Thanks for your quick answer.

Simon's answer did clarify it and I believe his solution is quite close to what 
you suggest and what is done in Qpack.


I'll try it out.


Thanks,

Eric


De : Simon Pfreundschuh 
Envoyé : vendredi 4 juin 2021 14:55:24
À : Sauvageat, Eric (IAP); arts_users...@mailman.rrz.uni-hamburg.de
Objet : Re: ARTS OEM multiple inversions


Sorry, though the response would automatically go to the list. For completeness,

here's my response from earlier.


Just to clarify and confirm:

> same sensors, same atmosphere, etc.. just different measurements vectors using
> OEM from ARTS directly ?

This is what ARTS currently does: It takes multiple observations and solves for 
a single atmospheric state. If I understand you correctly what you want to do is
same sensors but different atmospheres (although they may be the same a priori).

This kind of parallelism is not supported within ARTS so you'll have to do it
sequentially. It's straight forward when you use the Python interface: After
you have set up your calculation, you just call the OEM WSM multiple times in a 
loop.
What you need to pay attention to is to  reset the WSVs x, yf and jacobian 
before calling OEM a second time because those are both in- and outputs of the 
method.

I not sure, however, how much you will gain from this computationally since this
depends on your retrieval setup.

Let me know if anything remains unclear.

Kind regards,

Simon



From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de 
 on behalf of 
eric.sauvag...@iap.unibe.ch 
Sent: Friday, June 4, 2021 12:03:12 PM
To: arts_users...@mailman.rrz.uni-hamburg.de
Subject: [arts-users] ARTS OEM multiple inversions


Dear ARTS community,

I am performing strato-mesospheric ozone retrievals with the OEM implemented 
directly in ARTS (using the python interface).
As my sensor is not changing much with time, I am wondering if there is a way 
(and if this is worth in terms of computation speed) to reuse the 
sensor_response matrix (FFT channel response, unbinned) for different 
measurements which I think was quite straightforward with Qpack.

I was trying to write multiple calibrated spectra within the y vector as well 
as multiple meas. error cov. (Se and inv) blocks before running the OEM which I 
thought might be the way to go but it seems to treat them as multiple 
measurements of the same ozone profiles instead. Could somebody confirmed if 
this is expected and if yes, is there another way to do this kind of parallel 
OEM retrievals: same sensors, same atmosphere, etc.. just different 
measurements vectors using OEM from ARTS directly ?

Thanks for your help,
Eric


___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] ARTS OEM multiple inversions

[Simon Pfreundschuh]

Sorry, though the response would automatically go to the list. For completeness,

here's my response from earlier.


Just to clarify and confirm:

> same sensors, same atmosphere, etc.. just different measurements vectors using
> OEM from ARTS directly ?

This is what ARTS currently does: It takes multiple observations and solves for 
a single atmospheric state. If I understand you correctly what you want to do is
same sensors but different atmospheres (although they may be the same a priori).

This kind of parallelism is not supported within ARTS so you'll have to do it
sequentially. It's straight forward when you use the Python interface: After
you have set up your calculation, you just call the OEM WSM multiple times in a 
loop.
What you need to pay attention to is to  reset the WSVs x, yf and jacobian 
before calling OEM a second time because those are both in- and outputs of the 
method.

I not sure, however, how much you will gain from this computationally since this
depends on your retrieval setup.

Let me know if anything remains unclear.

Kind regards,

Simon



From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de 
 on behalf of 
eric.sauvag...@iap.unibe.ch 
Sent: Friday, June 4, 2021 12:03:12 PM
To: arts_users...@mailman.rrz.uni-hamburg.de
Subject: [arts-users] ARTS OEM multiple inversions


Dear ARTS community,

I am performing strato-mesospheric ozone retrievals with the OEM implemented 
directly in ARTS (using the python interface).
As my sensor is not changing much with time, I am wondering if there is a way 
(and if this is worth in terms of computation speed) to reuse the 
sensor_response matrix (FFT channel response, unbinned) for different 
measurements which I think was quite straightforward with Qpack.

I was trying to write multiple calibrated spectra within the y vector as well 
as multiple meas. error cov. (Se and inv) blocks before running the OEM which I 
thought might be the way to go but it seems to treat them as multiple 
measurements of the same ozone profiles instead. Could somebody confirmed if 
this is expected and if yes, is there another way to do this kind of parallel 
OEM retrievals: same sensors, same atmosphere, etc.. just different 
measurements vectors using OEM from ARTS directly ?

Thanks for your help,
Eric


___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] ARTS OEM multiple inversions

[Patrick Eriksson]

Eric,

I am performing strato-mesospheric ozone retrievals with the OEM 
implemented directly in ARTS (using the python interface).
As my sensor is not changing much with time, I am wondering if there is 
a way (and if this is worth in terms of computation speed) to reuse the 
sensor_response matrix (FFT channel response, unbinned) for different 
measurements which I think was quite straightforward with Qpack.


In Qpack some stuff is made behind the scene. Yes, it can be avoided to 
recalculate the sensor response. But you need save a number of 
variables. At least these ones:

sensor_response
sensor_response_f
sensor_response_pol
sensor_response_dlos,
sensor_response_f_grid, sensor_response_pol_grid,
sensor_response_dlos_grid

Maybe also good to save
mblock_dlos_grid
stokes_dim

And then you just need to import these for a new calculation (or just 
keep them in memory).



I was trying to write multiple calibrated spectra within the y vector as 
well as multiple meas. error cov. (Se and inv) blocks before running the 
OEM which I thought might be the way to go but it seems to treat them as 
multiple measurements of the same ozone profiles instead. Could somebody 
confirmed if this is expected and if yes, is there another way to do 
this kind of parallel OEM retrievals: same sensors, same atmosphere, 
etc.. just different measurements vectors using OEM from ARTS directly ?


There is no clear cut way inside ARTS to do batch OEM inversions. I am 
not using Python myself, but is this part not easy to setup on the 
python side?


Bye,

Patrick
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

[arts-users] ARTS OEM multiple inversions

[eric.sauvageat]

Dear ARTS community,

I am performing strato-mesospheric ozone retrievals with the OEM implemented 
directly in ARTS (using the python interface).
As my sensor is not changing much with time, I am wondering if there is a way 
(and if this is worth in terms of computation speed) to reuse the 
sensor_response matrix (FFT channel response, unbinned) for different 
measurements which I think was quite straightforward with Qpack.

I was trying to write multiple calibrated spectra within the y vector as well 
as multiple meas. error cov. (Se and inv) blocks before running the OEM which I 
thought might be the way to go but it seems to treat them as multiple 
measurements of the same ozone profiles instead. Could somebody confirmed if 
this is expected and if yes, is there another way to do this kind of parallel 
OEM retrievals: same sensors, same atmosphere, etc.. just different 
measurements vectors using OEM from ARTS directly ?

Thanks for your help,
Eric


___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi