Re: [GRASS-user] detrending time series maps

[Veronica Andreo via grass-user]

That's right, Ivan!
End of year mental death, sorry for the noise...

Vero

El vie, 29 dic 2023 a las 15:34, Ivan Marchesini ()
escribió:

> Hi veronica
>
>
> I see what you did with creating the days time series. In that way you
> acknowledge irregular gaps, right?
>
> yes this is the reason
>
>
> However, why do you multiply by days strds? From my understanding,
> detrending by subtracting the results of a model obbeys this rule: value(t)
> = observed(t) - predicted(t). Then, this
> mystrds-(regression_offset+regression_slope*days) should be
> mystrds-(regression_offset+regression_slope*mystrds).
>
> hmm
>
> may be I'm wrong but I used r.regression.series to assess a relationship
> between mystrds values and time (days)
>
> As a consequence the offset and slope maps I obtain are b and a in the
> following linear equation
>
> y=ax+b
>
> i.e.
>
> predicted_mystrds=slope*days+offset
>
> This is why I suppose that for detrending I need to do:
>
> mystrds-(predicted_mystrds)
>
> i.e.
>
> mystrds-(slope*days+offset)
>
>
> I'm I wrong?
>
> thank you
>
> Ivan
>
>
>
>
> Best,
> Vero
>
> El jue, 28 dic 2023 a las 8:14, Ivan Marchesini (<
> ivan.marches...@gmail.com>) escribió:
>
>> Dear Veronica
>>
>> I think I found a simple solution using temporal raster modules. Here is
>> an example:
>>
>> #evaluating info of the strds
>> eval `t.info mystrds -g`
>>
>> #getting the starting day (of the year, 0-365) of my strds
>> startday=$(date -d "$start_time" "+%j")
>>
>> #Creating a new strds where each pixel has the value of the count of the
>> days starting from the start_day of my strds (the start day in my dataset
>> is in 2016)
>> t.rast.mapcalc inputs=mystrds
>> expression="(start_year()-2016)*365-${startday} +start_doy()" output=days
>> basename=days nprocs=xxx --o
>>
>> #fitting the trend equation
>> r.regression.series xseries="`t.rast.list in=days columns=name sep=,
>> format=line`" yseries="` t.rast.list in=mystrds columns=name sep=,
>> format=line`"
>> out=regression_offset,regression_slope,regression_rsq,regression_t
>> meth=offset,slope,rsq,t
>>
>> #detrending
>> t.rast.mapcalc input=mystrds,days expression="mystrds_detrend
>> = mystrds-(regression_offset+regression_slope*days)" output=mystrds_detrend
>> basename=mystrds_detrend nprocs=xxx method=start --o
>>
>>
>> Best
>>
>> Ivan
>>
>>
>>
>>
>> On 23/12/23 14:53, Ivan Marchesini wrote:
>>
>> Hi Veronica
>>
>> Thank you. It goes in the direction of my idea evn if  my problem is
>> exactly trying to take into account the correct gaps between that data
>>
>> I have another idea.
>>
>> if it works I will come back here to explain how I did
>>
>> thank you again
>>
>> Ivan
>>
>>
>> On 22/12/23 13:45, Veronica Andreo wrote:
>>
>> Hello Ivan,
>>
>> AFAIU you could use the slope and offset maps from t.rast.series within
>> t.rast.algebra to detrend the values of the maps within the strds,
>> something like "detrended_strds = trend_strds - (trend_strds*map(slope) +
>> map(offset))". Others suggest, to detrend by subtracting the previous
>> value, i.e. that would imply using the temporal algebra with the temporal
>> index, something like "detrended_strds = trend_strds[1] - trend_strds[0]".
>>
>> I haven't tested any of these, just a couple of ideas ;-) However, I do
>> not know how this might interact with seasonality within data, or irregular
>> gaps.
>>
>> hth somehow
>> Vero
>>
>> El vie, 22 dic 2023 a las 5:10, Ivan Marchesini via grass-user (<
>> grass-user@lists.osgeo.org>) escribió:
>>
>>> Dear colleagues
>>>
>>> I would like to the advantage of the t.* modules for detrending a strd.
>>>
>>> In the strd I have earth observation data irregularly sampled (2 or 3
>>> times per month), in the period November-February, for 7 years. They are
>>> not equally spaced (i.e gaps have different duration)
>>>
>>> A simple t.rast.series analysis (opion=slope,offset) highlights that
>>> probably there is a descending trend when considering the maps ordered
>>> by id.
>>>
>>> I would like to fit a proper time depending fitting curve for each pixel
>>> and then subtract the function from the real data.
>>>
>>> any hints on how I can do this task exploiting the GRASS GIS modules or
>>> some simple bash/python scripting?
>>>
>>> thank you
>>>
>>> Ivan
>>>
>>>
>>>
>>>
>>> ___
>>> grass-user mailing list
>>> grass-user@lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/grass-user
>>>
>>
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] detrending time series maps

[Micha Silver via grass-user]

  
  
Hi Ivan:

  
On 29/12/2023 20:34, Ivan Marchesini
  via grass-user wrote:


  
  Hi veronica
  
  

  
  
  I see what you did with creating the days time series. In
that way you acknowledge irregular gaps, right? 

  
  yes this is the reason
  

  
  
  However, why do you multiply by days strds? From my
understanding, detrending by subtracting the results of a
model obbeys this rule: value(t) = observed(t) -
predicted(t). Then, this
mystrds-(regression_offset+regression_slope*days) should be
mystrds-(regression_offset+regression_slope*mystrds).

  
  hmm 
  
  may be I'm wrong but I used r.regression.series to assess a
relationship between mystrds values and time (days)
  
  As a consequence the offset and slope maps I obtain are b and a
in the following linear equation
  y=ax+b
  i.e.
  
  predicted_mystrds=slope*days+offset
  This is why I suppose that for detrending I need to do:
  
  mystrds-(predicted_mystrds)
  i.e.
  
  mystrds-(slope*days+offset)
  
  



I'm not sure about your procedure of regressing days from start
  of the time series vs the time series. But I would point out: 
The "elephant in the room" for any time series analysis is
  temporal autocorrelation.  If you have some seasonal variation in
  your data, then I don't think that such a regression could work.
  You mentioned data from Nov-Feb over several years, so I assume
  there is some seasonal effect going on?  Normally you have to
  first decompose the time series to remove the seasonal variation,
  and then examine the trend.





   
  I'm I wrong?
  thank you
  Ivan
  
  
  
  
  
  

  
  
  Best,
  Vero



  El jue, 28 dic 2023 a las
8:14, Ivan Marchesini ()
escribió:
  
  

  Dear Veronica
  I think I found a simple solution using temporal raster
modules. Here is an example:
  #evaluating info of the strds
eval `t.info mystrds -g`
  #getting the starting day (of the year, 0-365) of my
strds
startday=$(date -d "$start_time" "+%j")
  
  #Creating a new strds where each pixel has the value of
the count of the days starting from the start_day of my
strds (the start day in my dataset is in 2016)
t.rast.mapcalc inputs=mystrds
_expression_="(start_year()-2016)*365-${startday}
+start_doy()" output=days basename=days nprocs=xxx --o
  
  #fitting the trend equation
r.regression.series xseries="`t.rast.list in=days
columns=name sep=, format=line`" yseries="` t.rast.list
in=mystrds columns=name sep=, format=line`" 
out=regression_offset,regression_slope,regression_rsq,regression_t
meth=offset,slope,rsq,t
  
  #detrending
t.rast.mapcalc input=mystrds,days
_expression_="mystrds_detrend
= mystrds-(regression_offset+regression_slope*days)"
output=mystrds_detrend basename=mystrds_detrend
nprocs=xxx method=start --o
  
  
  
  Best
  Ivan
  
  
  
  
  
  
  
  On 23/12/23 14:53, Ivan Marchesini wrote:
  
  
Hi Veronica
Thank you. It goes in the direction of my idea evn
  if  my problem is exactly trying to take into account
  the correct gaps between that data 

I have another idea.
if it works I will come back here to explain how I
  did
thank you again
Ivan



On 22/12/23 13:45, Veronica Andreo wrote:


  Hello Ivan, 


AFAIU you could use the slope and offset maps
  from t.rast.series within t.rast.algebra to
  detrend the values of the maps within the strds,
  something like "detrended_strds = trend_strds -
  (trend_strds*map(slope) + map(offset))". Others
  suggest

Re: [GRASS-user] detrending time series maps

[Ivan Marchesini via grass-user]

Hi veronica



I see what you did with creating the days time series. In that way you 
acknowledge irregular gaps, right?

yes this is the reason


However, why do you multiply by days strds? From my understanding, 
detrending by subtracting the results of a model obbeys this rule: 
value(t) = observed(t) - predicted(t). Then, this 
mystrds-(regression_offset+regression_slope*days) should be 
mystrds-(regression_offset+regression_slope*mystrds).


hmm

may be I'm wrong but I used r.regression.series to assess a relationship 
between mystrds values and time (days)


As a consequence the offset and slope maps I obtain are b and a in the 
following linear equation


y=ax+b

i.e.

predicted_mystrds=slope*days+offset

This is why I suppose that for detrending I need to do:

mystrds-(predicted_mystrds)

i.e.

mystrds-(slope*days+offset)


I'm I wrong?

thank you

Ivan





Best,
Vero

El jue, 28 dic 2023 a las 8:14, Ivan Marchesini 
() escribió:


Dear Veronica

I think I found a simple solution using temporal raster modules.
Here is an example:

#evaluating info of the strds
eval `t.info  mystrds -g`

#getting the starting day (of the year, 0-365) of my strds
startday=$(date -d "$start_time" "+%j")

#Creating a new strds where each pixel has the value of the count
of the days starting from the start_day of my strds (the start day
in my dataset is in 2016)
t.rast.mapcalc inputs=mystrds
expression="(start_year()-2016)*365-${startday} +start_doy()"
output=days basename=days nprocs=xxx --o

#fitting the trend equation
r.regression.series xseries="`t.rast.list in=days columns=name
sep=, format=line`" yseries="` t.rast.list in=mystrds columns=name
sep=, format=line`"
out=regression_offset,regression_slope,regression_rsq,regression_t
meth=offset,slope,rsq,t

#detrending
t.rast.mapcalc input=mystrds,days expression="mystrds_detrend
= mystrds-(regression_offset+regression_slope*days)"
output=mystrds_detrend basename=mystrds_detrend nprocs=xxx
method=start --o


Best

Ivan




On 23/12/23 14:53, Ivan Marchesini wrote:


Hi Veronica

Thank you. It goes in the direction of my idea evn if my problem
is exactly trying to take into account the correct gaps between
that data

I have another idea.

if it works I will come back here to explain how I did

thank you again

Ivan


On 22/12/23 13:45, Veronica Andreo wrote:

Hello Ivan,

AFAIU you could use the slope and offset maps from t.rast.series
within t.rast.algebra to detrend the values of the maps within
the strds, something like "detrended_strds = trend_strds -
(trend_strds*map(slope) + map(offset))". Others suggest, to
detrend by subtracting the previous value, i.e. that would imply
using the temporal algebra with the temporal index, something
like "detrended_strds = trend_strds[1] - trend_strds[0]".

I haven't tested any of these, just a couple of ideas ;-)
However, I do not know how this might interact with seasonality
within data, or irregular gaps.

hth somehow
Vero

El vie, 22 dic 2023 a las 5:10, Ivan Marchesini via grass-user
() escribió:

Dear colleagues

I would like to the advantage of the t.* modules for
detrending a strd.

In the strd I have earth observation data irregularly
sampled (2 or 3
times per month), in the period November-February, for 7
years. They are
not equally spaced (i.e gaps have different duration)

A simple t.rast.series analysis (opion=slope,offset)
highlights that
probably there is a descending trend when considering the
maps ordered
by id.

I would like to fit a proper time depending fitting curve
for each pixel
and then subtract the function from the real data.

any hints on how I can do this task exploiting the GRASS GIS
modules or
some simple bash/python scripting?

thank you

Ivan




___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] detrending time series maps

[Veronica Andreo via grass-user]

Hello Ivan,

Thanks for coming back to this :)

I see what you did with creating the days time series. In that way you
acknowledge irregular gaps, right? Otherwise, as t.rast.series
method=slope,offset uses r.series in the background, it will use index as
independent variable and therefore maps are considered equally separated in
time.

However, why do you multiply by days strds? From my understanding,
detrending by subtracting the results of a model obbeys this rule: value(t)
= observed(t) - predicted(t). Then, this
mystrds-(regression_offset+regression_slope*days) should be
mystrds-(regression_offset+regression_slope*mystrds).

Best,
Vero

El jue, 28 dic 2023 a las 8:14, Ivan Marchesini ()
escribió:

> Dear Veronica
>
> I think I found a simple solution using temporal raster modules. Here is
> an example:
>
> #evaluating info of the strds
> eval `t.info mystrds -g`
>
> #getting the starting day (of the year, 0-365) of my strds
> startday=$(date -d "$start_time" "+%j")
>
> #Creating a new strds where each pixel has the value of the count of the
> days starting from the start_day of my strds (the start day in my dataset
> is in 2016)
> t.rast.mapcalc inputs=mystrds
> expression="(start_year()-2016)*365-${startday} +start_doy()" output=days
> basename=days nprocs=xxx --o
>
> #fitting the trend equation
> r.regression.series xseries="`t.rast.list in=days columns=name sep=,
> format=line`" yseries="` t.rast.list in=mystrds columns=name sep=,
> format=line`"
> out=regression_offset,regression_slope,regression_rsq,regression_t
> meth=offset,slope,rsq,t
>
> #detrending
> t.rast.mapcalc input=mystrds,days expression="mystrds_detrend
> = mystrds-(regression_offset+regression_slope*days)" output=mystrds_detrend
> basename=mystrds_detrend nprocs=xxx method=start --o
>
>
> Best
>
> Ivan
>
>
>
>
> On 23/12/23 14:53, Ivan Marchesini wrote:
>
> Hi Veronica
>
> Thank you. It goes in the direction of my idea evn if  my problem is
> exactly trying to take into account the correct gaps between that data
>
> I have another idea.
>
> if it works I will come back here to explain how I did
>
> thank you again
>
> Ivan
>
>
> On 22/12/23 13:45, Veronica Andreo wrote:
>
> Hello Ivan,
>
> AFAIU you could use the slope and offset maps from t.rast.series within
> t.rast.algebra to detrend the values of the maps within the strds,
> something like "detrended_strds = trend_strds - (trend_strds*map(slope) +
> map(offset))". Others suggest, to detrend by subtracting the previous
> value, i.e. that would imply using the temporal algebra with the temporal
> index, something like "detrended_strds = trend_strds[1] - trend_strds[0]".
>
> I haven't tested any of these, just a couple of ideas ;-) However, I do
> not know how this might interact with seasonality within data, or irregular
> gaps.
>
> hth somehow
> Vero
>
> El vie, 22 dic 2023 a las 5:10, Ivan Marchesini via grass-user (<
> grass-user@lists.osgeo.org>) escribió:
>
>> Dear colleagues
>>
>> I would like to the advantage of the t.* modules for detrending a strd.
>>
>> In the strd I have earth observation data irregularly sampled (2 or 3
>> times per month), in the period November-February, for 7 years. They are
>> not equally spaced (i.e gaps have different duration)
>>
>> A simple t.rast.series analysis (opion=slope,offset) highlights that
>> probably there is a descending trend when considering the maps ordered
>> by id.
>>
>> I would like to fit a proper time depending fitting curve for each pixel
>> and then subtract the function from the real data.
>>
>> any hints on how I can do this task exploiting the GRASS GIS modules or
>> some simple bash/python scripting?
>>
>> thank you
>>
>> Ivan
>>
>>
>>
>>
>> ___
>> grass-user mailing list
>> grass-user@lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/grass-user
>>
>
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] detrending time series maps

[Ivan Marchesini via grass-user]

Dear Veronica

I think I found a simple solution using temporal raster modules. Here is 
an example:


#evaluating info of the strds
eval `t.info mystrds -g`

#getting the starting day (of the year, 0-365) of my strds
startday=$(date -d "$start_time" "+%j")

#Creating a new strds where each pixel has the value of the count of the 
days starting from the start_day of my strds (the start day in my 
dataset is in 2016)
t.rast.mapcalc inputs=mystrds 
expression="(start_year()-2016)*365-${startday} +start_doy()" 
output=days basename=days nprocs=xxx --o


#fitting the trend equation
r.regression.series xseries="`t.rast.list in=days columns=name sep=, 
format=line`" yseries="` t.rast.list in=mystrds columns=name sep=, 
format=line`" 
out=regression_offset,regression_slope,regression_rsq,regression_t 
meth=offset,slope,rsq,t


#detrending
t.rast.mapcalc input=mystrds,days expression="mystrds_detrend 
= mystrds-(regression_offset+regression_slope*days)" 
output=mystrds_detrend basename=mystrds_detrend nprocs=xxx method=start --o



Best

Ivan




On 23/12/23 14:53, Ivan Marchesini wrote:


Hi Veronica

Thank you. It goes in the direction of my idea evn if  my problem is 
exactly trying to take into account the correct gaps between that data


I have another idea.

if it works I will come back here to explain how I did

thank you again

Ivan


On 22/12/23 13:45, Veronica Andreo wrote:

Hello Ivan,

AFAIU you could use the slope and offset maps from t.rast.series 
within t.rast.algebra to detrend the values of the maps within the 
strds, something like "detrended_strds = trend_strds - 
(trend_strds*map(slope) + map(offset))". Others suggest, to detrend 
by subtracting the previous value, i.e. that would imply using the 
temporal algebra with the temporal index, something like 
"detrended_strds = trend_strds[1] - trend_strds[0]".


I haven't tested any of these, just a couple of ideas ;-) However, I 
do not know how this might interact with seasonality within data, or 
irregular gaps.


hth somehow
Vero

El vie, 22 dic 2023 a las 5:10, Ivan Marchesini via grass-user 
() escribió:


Dear colleagues

I would like to the advantage of the t.* modules for detrending a
strd.

In the strd I have earth observation data irregularly sampled (2
or 3
times per month), in the period November-February, for 7 years.
They are
not equally spaced (i.e gaps have different duration)

A simple t.rast.series analysis (opion=slope,offset) highlights that
probably there is a descending trend when considering the maps
ordered
by id.

I would like to fit a proper time depending fitting curve for
each pixel
and then subtract the function from the real data.

any hints on how I can do this task exploiting the GRASS GIS
modules or
some simple bash/python scripting?

thank you

Ivan




___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] detrending time series maps

[Ivan Marchesini via grass-user]

Hi Veronica

Thank you. It goes in the direction of my idea evn if  my problem is 
exactly trying to take into account the correct gaps between that data


I have another idea.

if it works I will come back here to explain how I did

thank you again

Ivan


On 22/12/23 13:45, Veronica Andreo wrote:

Hello Ivan,

AFAIU you could use the slope and offset maps from t.rast.series 
within t.rast.algebra to detrend the values of the maps within the 
strds, something like "detrended_strds = trend_strds - 
(trend_strds*map(slope) + map(offset))". Others suggest, to detrend by 
subtracting the previous value, i.e. that would imply using the 
temporal algebra with the temporal index, something like 
"detrended_strds = trend_strds[1] - trend_strds[0]".


I haven't tested any of these, just a couple of ideas ;-) However, I 
do not know how this might interact with seasonality within data, or 
irregular gaps.


hth somehow
Vero

El vie, 22 dic 2023 a las 5:10, Ivan Marchesini via grass-user 
() escribió:


Dear colleagues

I would like to the advantage of the t.* modules for detrending a
strd.

In the strd I have earth observation data irregularly sampled (2 or 3
times per month), in the period November-February, for 7 years.
They are
not equally spaced (i.e gaps have different duration)

A simple t.rast.series analysis (opion=slope,offset) highlights that
probably there is a descending trend when considering the maps
ordered
by id.

I would like to fit a proper time depending fitting curve for each
pixel
and then subtract the function from the real data.

any hints on how I can do this task exploiting the GRASS GIS
modules or
some simple bash/python scripting?

thank you

Ivan




___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] detrending time series maps

[Veronica Andreo via grass-user]

Hello Ivan,

AFAIU you could use the slope and offset maps from t.rast.series within
t.rast.algebra to detrend the values of the maps within the strds,
something like "detrended_strds = trend_strds - (trend_strds*map(slope) +
map(offset))". Others suggest, to detrend by subtracting the previous
value, i.e. that would imply using the temporal algebra with the temporal
index, something like "detrended_strds = trend_strds[1] - trend_strds[0]".

I haven't tested any of these, just a couple of ideas ;-) However, I do not
know how this might interact with seasonality within data, or irregular
gaps.

hth somehow
Vero

El vie, 22 dic 2023 a las 5:10, Ivan Marchesini via grass-user (<
grass-user@lists.osgeo.org>) escribió:

> Dear colleagues
>
> I would like to the advantage of the t.* modules for detrending a strd.
>
> In the strd I have earth observation data irregularly sampled (2 or 3
> times per month), in the period November-February, for 7 years. They are
> not equally spaced (i.e gaps have different duration)
>
> A simple t.rast.series analysis (opion=slope,offset) highlights that
> probably there is a descending trend when considering the maps ordered
> by id.
>
> I would like to fit a proper time depending fitting curve for each pixel
> and then subtract the function from the real data.
>
> any hints on how I can do this task exploiting the GRASS GIS modules or
> some simple bash/python scripting?
>
> thank you
>
> Ivan
>
>
>
>
> ___
> grass-user mailing list
> grass-user@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-user
>
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

[GRASS-user] detrending time series maps

[Ivan Marchesini via grass-user]

Dear colleagues

I would like to the advantage of the t.* modules for detrending a strd.

In the strd I have earth observation data irregularly sampled (2 or 3 
times per month), in the period November-February, for 7 years. They are 
not equally spaced (i.e gaps have different duration)


A simple t.rast.series analysis (opion=slope,offset) highlights that 
probably there is a descending trend when considering the maps ordered 
by id.


I would like to fit a proper time depending fitting curve for each pixel 
and then subtract the function from the real data.


any hints on how I can do this task exploiting the GRASS GIS modules or 
some simple bash/python scripting?


thank you

Ivan




___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user