Re: [arts-users] ARTS OEM

2018-09-17 Thread 이병석
Hello Richard,
​
Thank you for your reply. 
​
May I ask which tips xml files you are referring to?
​
Also a question: it seems that the bad partition function warning occurs for 
any input t_field temperature beyond the range between 150 K and 300 K. In 
reality the surface temperature of the Earth frequently exceeds 300 K (26.85 
deg C). For input values such as 30 deg C or a little higher (e.g. 30-40 deg C, 
early 300's K), would the results still be bad if the warning is suppressed?
​
Regards,
​
Byungsuk Lee
ARA Consulting & Technology
30 Songdomirae-ro D-1510, Yeonsu-gu, Incheon 21990, South Korea

-Original Message-
From: "Richard Larsson"
To: "이병석";
Cc: "Simon Pfreundschuh"; 
;
Sent: 2018-09-17 (월) 18:21:57
Subject: Re: [arts-users] ARTS OEM
 
Hi,
 
Just as a warning, suppressing the bad partition function error is not good 
practice since it is known that they are, you know, bad.  Standard tests in 
ARTS fail by several tenths of Kelvin because the partition functions in ARTS 
are that bad.  Just run with the provided tips xml-files instead, because if 
your temperatures are lower than 70 K or higher than 3000 K you probably have 
another problem.
 
With hope,
//Richard

2018-09-17 10:31 GMT+02:00 이병석 :

Hello Simon Pfreundschuh,
​
Thank you for your reply.
​
I have employed both of the suggestions and still get the irregular errors. 
From the same controlfile modified in the previous email, I have included and 
changed to the following lines:
​
OEM(method="lm",
   max_iter=15,
   display_progress=1,
   lm_ga_settings=[100,5,2,1000,1,99])
​
and
​
AgendaSet( inversion_iterate_agenda ){
 
 xClip(ijq = 0, limit_low = 150.0, limit_high = 350)
​
 Ignore(inversion_iteration_counter)
 # Map x to ARTS' variables
 x2artsStandard
​
 # To be safe, rerun checks dealing with the atmosphere
 atmfields_checkedCalc
 atmgeom_checkedCalc
​
 # Calculate yf and Jacobian matching x.
 yCalc( y=yf )
​
 # Add baseline term
 VectorAddVector( yf, yf, y_baseline )
​
 # This method takes cares of some "fixes" that are needed to get the Jacobian
 # right for iterative solutions. No need to call this WSM for linear 
inversions.
 jacobianAdjustAndTransform
}
​
I included both of the suggestions, and from there increased the maximum number 
of iterations to 15 and set the temperature limits to be between 150 K and 350 
K. All other lines remain the same, including the input absorption line and 
atmospheric fields data. 
​​​
Now I irregularly get the errors during the OEM computation as before, of 
t_field having NaN values or of bad partition function warning (which happens 
for any t_field value outside between 150 K and 300 K). While the bad partition 
function warning could be suppressed in the code, it seems that the same 
problem as before persists. 
​
Could you please look into this issue a little more? I am not sure how to 
tackle the problem. 
​
Thank you for your service.
​
Regards, 
​
Byungsuk Lee
ARA Consulting & Technology
30 Songdomirae-ro D-1510, Yeonsu-gu, Incheon 21990, South Korea
 

-Original Message-
From: "Simon Pfreundschuh"
To: "이병석"; 
"arts_users...@mailman.rrz.uni-hamburg.de";
Cc:
Sent: 2018-09-14 (금) 15:13:55
Subject: Re: [arts-users] ARTS OEM
 
Hi Byungsuk Lee,

It is a bit surprising that these errors occur irregularly, since there is no 
randomness
in the test setup. However, if you are running retrievals with external 
observation data
it may well happen that the iteration ends up in a state that produces invalid 
values.

For your temperature retrieval I would recommend one or both of the following 
solutions:

1. Use the Levenberg-Marquardt (LM) iteration scheme: Using the LM scheme with 
a high start
   value for the gamma parameter can help avoid reaching invalid values when 
the forward
   model is non-linear:

  OEM(method="lm",
  max_iter=5,
  display_progress=1,
  lm_ga_settings=[100,5,2,1000,1,99])

2. Clip the x-vector: Using the xClip WSM inside your inversion iterate agenda 
you can force the
   x vector to contain only temperatures within a certain range. If you use 
this approach you
   should check that none of the retrieved temperatures in your final state are 
clipped otherwise your
   results may not be valid.

AgendaSet( inversion_iterate_agenda ){

  xClip(ijq = 0, limit_low = 100.0)
  Ignore(inversion_iteration_counter)
  # Map x to ARTS' variables
  x2artsStandard

  # To be safe, rerun checks dealing with the atmosphere
  atmfields_checkedCalc
  atmgeom_checkedCalc

  # Calculate yf and Jacobian matching x.
  yCalc( y=yf )

  # Add baseline term
  VectorAddVector( yf, yf, y_baseline )

  # This method takes cares of some "fixes" that are needed to get the 
Jacobian
  # right for iterative solutions. No need to call this WSM for linear 
inversions.
  jacobianAdjustAndTransform
}


Regards,

Simon

From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de 
 on behalf of 이병석 


Re: [arts-users] ARTS OEM

2018-09-17 Thread Richard Larsson
Hi,

Just as a warning, suppressing the bad partition function error is not good
practice since it is known that they are, you know, bad.  Standard tests in
ARTS fail by several tenths of Kelvin because the partition functions in
ARTS are that bad.  Just run with the provided tips xml-files instead,
because if your temperatures are lower than 70 K or higher than 3000 K you
probably have another problem.

With hope,
//Richard

2018-09-17 10:31 GMT+02:00 이병석 :

> Hello Simon Pfreundschuh,
> ​
> Thank you for your reply.
> ​
> I have employed both of the suggestions and still get the irregular
> errors. From the same controlfile modified in the previous email, I have
> included and changed to the following lines:
> ​
> OEM(method="lm",
>max_iter=15,
>display_progress=1,
>lm_ga_settings=[100,5,2,1000,1,99])
> ​
> and
> ​
> AgendaSet( inversion_iterate_agenda ){
>
>  xClip(ijq = 0, limit_low = 150.0, limit_high = 350)
> ​
>  Ignore(inversion_iteration_counter)
>  # Map x to ARTS' variables
>  x2artsStandard
> ​
>  # To be safe, rerun checks dealing with the atmosphere
>  atmfields_checkedCalc
>  atmgeom_checkedCalc
> ​
>  # Calculate yf and Jacobian matching x.
>  yCalc( y=yf )
> ​
>  # Add baseline term
>  VectorAddVector( yf, yf, y_baseline )
> ​
>  # This method takes cares of some "fixes" that are needed to get the
> Jacobian
>  # right for iterative solutions. No need to call this WSM for linear
> inversions.
>  jacobianAdjustAndTransform
> }
> ​
> I included both of the suggestions, and from there
> increased the maximum number of iterations to 15 and set the temperature 
> limits to be between 150
> K and 350 K. All other lines remain the same, including the input
> absorption line and atmospheric fields data.
> ​​​
> Now I irregularly get the errors during the OEM computation as before, of
> t_field having NaN values or of bad partition function warning (which
> happens for any t_field value outside between 150 K and 300 K). While the
> bad partition function warning could be suppressed in the code, it seems
> that the same problem as before persists.
> ​
> Could you please look into this issue a little more? I am not sure how to
> tackle the problem.
> ​
> Thank you for your service.
> ​
> Regards,
> ​
>
> Byungsuk Lee
>
> ARA Consulting & Technology
>
> 30 Songdomirae-ro D-1510, Yeonsu-gu, Incheon 21990, South Korea
>
>
>
> -Original Message-
> *From:* "Simon Pfreundschuh"
> *To:* "이병석"; "arts_users...@mailman.rrz.uni-hamburg.de"<
> arts_users...@mailman.rrz.uni-hamburg.de>;
> *Cc:*
> *Sent:* 2018-09-14 (금) 15:13:55
> *Subject:* Re: [arts-users] ARTS OEM
>
> Hi Byungsuk Lee,
>
> It is a bit surprising that these errors occur irregularly, since there is
> no randomness
> in the test setup. However, if you are running retrievals with external
> observation data
> it may well happen that the iteration ends up in a state that produces
> invalid values.
>
> For your temperature retrieval I would recommend one or both of the
> following solutions:
>
> 1. Use the Levenberg-Marquardt (LM) iteration scheme: Using the LM scheme
> with a high start
>value for the gamma parameter can help avoid reaching invalid values
> when the forward
>model is non-linear:
>
>   OEM(method="lm",
>   max_iter=5,
>   display_progress=1,
>   lm_ga_settings=[100,5,2,1000,1,99])
>
> 2. Clip the x-vector: Using the xClip WSM inside your inversion iterate
> agenda you can force the
>x vector to contain only temperatures within a certain range. If you
> use this approach you
>should check that none of the retrieved temperatures in your final
> state are clipped otherwise your
>results may not be valid.
>
> AgendaSet( inversion_iterate_agenda ){
>
>   xClip(ijq = 0, limit_low = 100.0)
>   Ignore(inversion_iteration_counter)
>   # Map x to ARTS' variables
>   x2artsStandard
>
>   # To be safe, rerun checks dealing with the atmosphere
>   atmfields_checkedCalc
>   atmgeom_checkedCalc
>
>   # Calculate yf and Jacobian matching x.
>   yCalc( y=yf )
>
>   # Add baseline term
>   VectorAddVector( yf, yf, y_baseline )
>
>   # This method takes cares of some "fixes" that are needed to get the
> Jacobian
>   # right for iterative solutions. No need to call this WSM for linear
> inversions.
>   jacobianAdjustAndTransform
> }
>
>
> Regards,
>
> Simon
>
> --
> *From:* arts_users.mi-boun...@mailman.rrz.uni-hamburg.de <
> arts_users.mi-boun...@mailman.rrz.uni-hamburg.de> on behalf of 이병석 <
> cb...@aracnt.com>
> *Sent:* Friday, September 14, 2018 2:31:25 AM
> *To:* arts_users...@mailman.rrz.uni-hamburg.de
> *Subject:* [arts-users] ARTS OEM
>
> Dear ARTS users,
> ​
> I am writing to address a question with the use of OEM in the ARTS
> development version.
> ​
> From the "TestOEM.arts" in "ARTS/controlfiles/artscomponents/oem", this
> time I changed the codes as follows.
> ​
> 1) From (line 

Re: [arts-users] ARTS OEM

2018-09-17 Thread 이병석
Hello Simon Pfreundschuh,
​
Thank you for your reply.
​
I have employed both of the suggestions and still get the irregular errors. 
From the same controlfile modified in the previous email, I have included and 
changed to the following lines:
​
OEM(method="lm",
   max_iter=15,
   display_progress=1,
   lm_ga_settings=[100,5,2,1000,1,99])
​
and
​
AgendaSet( inversion_iterate_agenda ){
 
 xClip(ijq = 0, limit_low = 150.0, limit_high = 350)
​
 Ignore(inversion_iteration_counter)
 # Map x to ARTS' variables
 x2artsStandard
​
 # To be safe, rerun checks dealing with the atmosphere
 atmfields_checkedCalc
 atmgeom_checkedCalc
​
 # Calculate yf and Jacobian matching x.
 yCalc( y=yf )
​
 # Add baseline term
 VectorAddVector( yf, yf, y_baseline )
​
 # This method takes cares of some "fixes" that are needed to get the Jacobian
 # right for iterative solutions. No need to call this WSM for linear 
inversions.
 jacobianAdjustAndTransform
}
​
I included both of the suggestions, and from there increased the maximum number 
of iterations to 15 and set the temperature limits to be between 150 K and 350 
K. All other lines remain the same, including the input absorption line and 
atmospheric fields data. 
​​​
Now I irregularly get the errors during the OEM computation as before, of 
t_field having NaN values or of bad partition function warning (which happens 
for any t_field value outside between 150 K and 300 K). While the bad partition 
function warning could be suppressed in the code, it seems that the same 
problem as before persists. 
​
Could you please look into this issue a little more? I am not sure how to 
tackle the problem. 
​
Thank you for your service.
​
Regards, 
​
Byungsuk Lee
ARA Consulting & Technology
30 Songdomirae-ro D-1510, Yeonsu-gu, Incheon 21990, South Korea


-Original Message-
From: "Simon Pfreundschuh"
To: "이병석"; 
"arts_users...@mailman.rrz.uni-hamburg.de";
Cc:
Sent: 2018-09-14 (금) 15:13:55
Subject: Re: [arts-users] ARTS OEM
 
Hi Byungsuk Lee,

It is a bit surprising that these errors occur irregularly, since there is no 
randomness
in the test setup. However, if you are running retrievals with external 
observation data
it may well happen that the iteration ends up in a state that produces invalid 
values.

For your temperature retrieval I would recommend one or both of the following 
solutions:

1. Use the Levenberg-Marquardt (LM) iteration scheme: Using the LM scheme with 
a high start
   value for the gamma parameter can help avoid reaching invalid values when 
the forward
   model is non-linear:

  OEM(method="lm",
  max_iter=5,
  display_progress=1,
  lm_ga_settings=[100,5,2,1000,1,99])

2. Clip the x-vector: Using the xClip WSM inside your inversion iterate agenda 
you can force the
   x vector to contain only temperatures within a certain range. If you use 
this approach you
   should check that none of the retrieved temperatures in your final state are 
clipped otherwise your
   results may not be valid.

AgendaSet( inversion_iterate_agenda ){

  xClip(ijq = 0, limit_low = 100.0)
  Ignore(inversion_iteration_counter)
  # Map x to ARTS' variables
  x2artsStandard

  # To be safe, rerun checks dealing with the atmosphere
  atmfields_checkedCalc
  atmgeom_checkedCalc

  # Calculate yf and Jacobian matching x.
  yCalc( y=yf )

  # Add baseline term
  VectorAddVector( yf, yf, y_baseline )

  # This method takes cares of some "fixes" that are needed to get the 
Jacobian
  # right for iterative solutions. No need to call this WSM for linear 
inversions.
  jacobianAdjustAndTransform
}


Regards,

Simon

From: arts_users.mi-boun...@mailman.rrz.uni-hamburg.de 
 on behalf of 이병석 

Sent: Friday, September 14, 2018 2:31:25 AM
To: arts_users...@mailman.rrz.uni-hamburg.de
Subject: [arts-users] ARTS OEM
 
Dear ARTS users,
​
I am writing to address a question with the use of OEM in the ARTS development 
version.  
​
From the "TestOEM.arts" in "ARTS/controlfiles/artscomponents/oem", this time I 
changed the codes as follows.
​
1) From (line 115):
retrievalAddAbsSpecies(
species = "O3",
unit = "vmr",
g1 = p_ret_grid,
g2 = lat_grid,
g3 = lon_grid
)
To:
retrievalAddTemperature (
  g1 = p_ret_grid,
  g2 = lat_grid,
  g3 = lon_grid,
  hse = "off"   )
​
2) From (line 125):
VectorSetConstant(vars, nelem, 1e-12)
DiagonalMatrix(sparse_block, vars)
covmat_sxAddBlock(block = sparse_block)
To:
VectorSetConstant(vars, nelem, 0.1)
DiagonalMatrix(sparse_block, vars)
covmat_sxAddBlock(block = sparse_block)
​
3) From (line 37): 
NumericSet( f_start, 110.436e9 )
NumericSet( f_end, 111.236e9 )
IndexSet( nf, 801 )
IndexSet( np, 81 )
To:
NumericSet( f_start, 110.436e9 )
NumericSet( f_end, 111.236e9 )
IndexSet( nf, 18 )
IndexSet( np, 18 )
​
4) From (line 42): 
VectorNLinSpace( f_grid, nf, f_start, f_end )
VectorNLogSpace( p_grid,361, 500e2, 0.1 )
VectorNLogSpace( p_ret_grid, np, 500e2, 0.1 )
To: