Re: [arts-users] Advise concerning ARTS computation speed

[Pauline Martinet]

Hi everybody,

I tried to generate an absorption look-up-table from Qarts following the 
example in demos/qarts_abstable_demo.m

When I call arts_y, I get the error message described later in the email.

Does anyone know what I should modify to make it work ?

I also generated an .xml look-up-table from the arts controfile directory with 
the good frequency grid and
absorption species, but I am still investigating how to use it directly in a 
control file or
in Qarts.

Thanks a lot,

Best regards

Pauline


Run-time error in controlfile: 
/tmp/atmlab-martinet-tped3b3242_9737_421a_8c1f_5eccc8a34610/cfile.arts
Run-time error in method: propmat_clearsky_fieldCalc
Run-time error in agenda: propmat_clearsky_agenda
Run-time error in method: propmat_clearskyAddFromLookup
Problem with gas absorption lookup table.
Temperature T is outside the range covered by the lookup table.
Your temperature was 215.506 K at a pressure of 2596.02 Pa.
The temperature offset value is -61.4155.
The allowed range is -17.1429 to 17.1429.
The temperature perturbation grid range in the table is -15 to 15.
We allow a bit of extrapolation, but NOT SO MUCH!
Stopping ARTS execution.
Goodbye.





- Météo-France - 
Dr. Pauline Martinet 
Chercheur CNRM/GMEI/LISA 
pauline.marti...@meteo.fr 
Fixe : +33 561079031 

- Mail original -
De: "Patrick Eriksson" <patrick.eriks...@chalmers.se>
À: "Pauline Martinet" <pauline.marti...@meteo.fr>, "Oliver Lemke" 
<oliver.le...@uni-hamburg.de>
Cc: "francesco deangelis1" <francesco.deangel...@graduate.univaq.it>, "ARTS 
Users List" <arts_users.mi@lists.uni-hamburg.de>, "domenico cimini" 
<domenico.cim...@imaa.cnr.it>
Envoyé: Mercredi 13 Juillet 2016 11:01:00
Objet: Re: [arts-users] Advise concerning ARTS computation speed

Hi again,

> I will see what I can find in the users guide to prepare a look-up-table for 
> my instrument.

How to practically generate a look-up table is best described by demo 
scripts. For pure arts see

controlfiles/artscomponents/absorption/TestAbs.arts

For Qarts see

demos/qarts_abstable_demo.m

/Patrick
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] Advise concerning ARTS computation speed

[Patrick Eriksson]

Hi again,


I will see what I can find in the users guide to prepare a look-up-table for my 
instrument.


How to practically generate a look-up table is best described by demo 
scripts. For pure arts see


controlfiles/artscomponents/absorption/TestAbs.arts

For Qarts see

demos/qarts_abstable_demo.m

/Patrick
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] Advise concerning ARTS computation speed

[Patrick Eriksson]

Dear Pauline,

Just browsed through the email conversion. Yes, I missed the debug 
compilation option, that Richard and Oliver clarified.


Seems that others have answered most things. But I need to comment one 
thing.




I already use the "on-the-fly" option to compute the absorption which
should be less time consuming than the look-up table (from Patrick's
advice). Patrick also mentioned that nothing should be critical if we
use standard setting.


I probably expressed myself poorly. In fact I think your simulations 
should be faster with a lookup table. On-the-fly could be faster for 
ground-based radiometers, but I had then in mind ones targeting the 
strato- and mesosphere. For such radiometers you only need to consider a 
single transition (neglecting the issue of tropospheric correction).


I assume you are running full PWR or MPM models, and then a lookup table 
should be faster. But this is just an estimate, and I can not say if the 
difference is small or large.


Regards,

Patrick
___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] Advise concerning ARTS computation speed

[Oliver Lemke]

Hi Pauline,

If you didn't pass any options to cmake at the time you compiled ARTS, then you 
have an executable with debug information because that's the default.

To get a non-debug build you have to explicitly set it to release mode as 
Richard explained:

cmake -DCMAKE_BUILD_TYPE=Release ..
make clean
make

cheers,
/oliver


> On 12 Jul 2016, at 17:29, Pauline Martinet  wrote:
> 
> Hi Richard,
> 
> Thanks for your advice.
> 
> I am using the debug mode only for atmlab (the matlab interface to call ARTS).
> 
> I do not think I compiled ARTS in any debug mode. Thus, I assume it should 
> not have an impact on the ARTS speed.
> 
> Best regards,
> 
> Pauline

___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] Advise concerning ARTS computation speed

[Pauline Martinet]

Dear Patrick and Stephan,

Thanks a lot for your kind help.

I managed to run ARTS with atmlab and a timer (Patrick's suggestion) as well as 
run ARTS directly
from the cfile created by ATMLAB (with a Debug mode). I assume this way we have 
a fair evaluation of
the computation time of the ARTS simulation.

I already use the "on-the-fly" option to compute the absorption which should be 
less time consuming
than the look-up table (from Patrick's advice). Patrick also mentioned that 
nothing should be critical if we use standard setting.

Stephan, do you have any example of not standard setting (to check if I have an 
option which increases the computation time of ARTS).

I also was successful to compute the Jacobians directly from an arts 
controlfile. But when I want to copy the output into a matrix
to write the values in an ASCII file (following the example TestWfuns.arts: 
Copy( Ja, jacobian ) I get the error message:
Method Copy needs input variable: jacobian

I will have a closer look to the users guide and I let you know what is unclear.

Thanks a lot again,

Best regards,

Pauline

- Météo-France - 
Dr. Pauline Martinet 
Chercheur CNRM/GMEI/LISA 
pauline.marti...@meteo.fr 
Fixe : +33 561079031 

- Mail original -
De: "Stefan Buehler" 
À: "Pauline Martinet" 
Cc: "Patrick Eriksson" , "domenico cimini" 
, "francesco deangelis1" 
, "arts users mi" 

Envoyé: Lundi 11 Juillet 2016 17:58:37
Objet: Re: Advise concerning ARTS computation speed

Dear Pauline,

> Could anyone help in explaining what is the function to compute Jacobians 
> from ARTS alone with atmlab ?

There are several example ARTS controlfiles with Jacobians included in the 
distribution, e.g.:
controlfiles/artscomponents/wfuns/TestWfuns.arts
controlfiles/classroom_exercises/exe3_jacobian/jacobian.arts

You find documentation on each individual ARTS method on the docserver:
http://arts.mi.uni-hamburg.de/docserver-trunk/
or if you start the docserver that comes with your ARTS distribution.

> When I have the cfile from ARTS, what is the terminal command to execute it 
> without any matlab interface ?

arts jacobian.arts

If your arts executable is not in your shell path, you have to give the right 
path, e.g.,

~/arts_build/src/arts jacobian.arts

jacobian.arts here is the name of the controlfile.

> (it is probably a silly question  but I really had troubles to understand how 
> to use ARTS without
> matlab this is why I never used).

Have you tried the ARTS user guide (available at radiativetransfer.org)? How to 
run ARTS is described in Chapter 1. Please let us know if anything there is 
unclear, so that we can improve the documentation.

Running ARTS without Matlab is really easier (and more thoroughly documented) 
than through the Matlab interface. (But so far lacking the capabilities to run 
retrievals, of course.)

Best wishes,

Stefan

___
arts_users.mi mailing list
arts_users.mi@lists.uni-hamburg.de
https://mailman.rrz.uni-hamburg.de/mailman/listinfo/arts_users.mi

Re: [arts-users] Advise concerning ARTS computation speed

[Patrick Eriksson]

Hi Pauline,

Some quick input, I really hope others can provide more detailed help.

A small example on how to define the jacobian in ARTS:

jacobianInit
jacobianAddAbsSpecies( jacobian_quantities, jacobian_agenda,
atmosphere_dim, p_grid, lat_grid, lon_grid,
retrieval_grid, lat_grid, lon_grid,
"O3", "analytical", "rel", 1, 0.01 )
jacobianClose

This example just considers ozone. You can make more "jacobianAdd" to 
also include other quantities.


The actual calculation is done by yCalc.

If you have a control file, you run it as

arts cfile.arts

For help do

arts -h

For example, to control screen output

arts -r22 cfile.arts

Just a quick into. If you have further questions, others have to answer.

Bye,

Patrick



On 07/11/16 17:24, Pauline Martinet wrote:

Hi Patrick (cc. arts users),

Thanks for your reply.

We use the on-the-fly option and not the lookup table to compute the
absorption.

Could anyone help in explaining what is the function to compute
Jacobians from ARTS alone with atmlab ?

When I have the cfile from ARTS, what is the terminal command to
execute it without any matlab interface ?

(it is probably a silly question  but I really had troubles to
understand how to use ARTS without matlab this is why I never used).

Thanks a lot for your kind help,

Best regards,

Pauline

- Météo-France - Dr. Pauline Martinet Chercheur
CNRM/GMEI/LISA pauline.marti...@meteo.fr Fixe : +33 561079031

- Mail original - De: "Patrick Eriksson"
 À: "domenico cimini"
 Cc: "Stefan Buehler"
, "pauline martinet"
, "francesco deangelis1"
, "arts users mi"
 Envoyé: Lundi 11 Juillet 2016
17:13:10 Objet: Re: Advise concerning ARTS computation speed

Dear Nico,

A relatively quick answer as I am on vacation. Please send questions
like this to the arts users mailing list. See cc.

As I understand it, this deals with the relative speed of ARTS and
RTTOV. I personally don't have any practical experience of RTTOV.

Unfortunately, there is no single recommendation on how to use ARTS
for maximum speed. The most critical aspect is how the absorption is
calculated, on-the-fly or by lookup table. On-the-fly can be the
fastest option, particularly for ground-based measurement of the
strato- and mesosphere. I assume you should use the lookup table
option. If you allow some poorer accuracy, the polynomial order used
to interpolate the lookup table could be decreased to increase the
speed. If you use standard settings, other aspects should not be
critical. It would be lengthy discuss all possible bad settings, ask
if you are unsure about any setting.

You don't need Qpack to calculate Jacobians. This can be done by
ARTS alone. However, it should be possible to time the actual time
used by ARTS to calculate the spectrum + Jacobian, even from Qpack.

Try this

Q.YCALC_WSMS = { 'timerStart', 'yCalc', 'timerStop' };

You could need to set this

atmlab('FMODEL_VERBOSITY',1)

But all this could fail, we have experienced problems with how
matlab picks up the screen output of arts.

For more advanced qpack users, one option is to set

atmlab('DEBUG',1)

and locate the relavnt temporary folder, add the time commands to
the cfile and run the cfile from the terminal.

Regards,

Patrick











*Från:* "domenico.cimini" 

*Skickat:* 8 juli 2016 10:54:18 CEST *Till:* Stefan Buehler
, patrick.eriks...@chalmers.se *Cc:*
Pauline Martinet , Francesco De Angelis
 *Ämne:* Advise concerning
ARTS computation speed

Dear Stefan and Patrick,

let us intrude in your mailbox for asking your advise concerning
ARTS.

Some months ago we submitted a manuscript presenting the modification
we made to RTTOV to simulate ground-based microwave radiometer
observations: http://www.geosci-model-dev-discuss.net/gmd-2016-65/

As we compare results from RTTOV and ARTS, we also reported on the
different computing times.

Referee #2 commented that the speed of ARTS depends critically on
several settings. For details, see the comment at the bottom of C3 in
the following link:
http://editor.copernicus.org/index.php/gmd-2016-65-RC2.pdf?_mdl=msover_md&_jrl=365&_lcm=oc108lcm109w&_acm=get_comm_file&_ms=50514=108717=1176682561237999184

 Of course we want to be fair with ARTS, but we may miss the
necessary familiarity with ARTS settings; thus we would like to ask
your advise on the following issues.

1. Do you have any feeling of the computation speed of ARTS with
respect to RTTOV?

2. We used ARTS + ATMLAB for the direct simulations (through the
arts_y(Q) function) and ARTS + Qpack2 to calculate and evaluate the
computation time of the Jacobians.