Re: [R] aggregation of irregular interval time-series
Hello
thanks to my friend Diego, a computer agronomist,
I was able to figure out how to do it.
I still have only one problem: how do I calculate the data for the
first week? The traps were set up on the fields on May 5th. On that
date, a 0 could be entered for all time-series. But how to do it with a
simple command?
> # the data (as copied from kobotoolbox, with the datetime of each
> capture observation, and the farm (azienda) and type of trap
> (trappola)
> library(data.table)
> library(parsedate)
> vindicta_catture_cimici<-data.table::data.table(
> datetime = parse_iso_8601(c("2021-05-29 17:00:00",
> "2021-06-03 09:20:00","2021-06-05 10:52:00","2021-
> 06-03 09:30:00",
> "2021-06-01 17:00:00","2021-05-16 08:33:00","2021-
> 05-16 08:34:00",
> "2021-06-05 17:00:00","2021-06-05 11:00:00","2021-
> 05-23 18:00:00",
> "2021-06-03 08:45:00","2021-06-01 07:30:00",
> "2021-05-29 05:30:00","2021-05-30 15:00:00","2021-
> 05-29 06:00:00",
> "2021-05-23 06:30:00","2021-05-20 13:00:00","2021-
> 05-20 13:15:00",
> "2021-05-20 12:45:00","2021-05-16 16:15:00","2021-
> 05-16 16:30:00",
> "2021-05-20 13:00:00","2021-05-15 12:09:00",
> "2021-06-11 15:27:00","2021-06-11 15:30:00","2021-
> 06-11 15:43:00",
> "2021-06-11 15:50:00","2021-06-12 19:01:00")),
> catture = c(25,0,92,0,23,0,2,5,0,0,6,0,30,
> 43,0,23,3,0,0,0,13,4,15,100,0,15,2,20),
> azienda = as.factor(c("arvaia","toderici",
>"arvaia","toderici","arvaia","arvaia","arv
> aia",
>"arvaia","arvaia","arvaia","toderici","mag
> li",
>"arvaia","magli","arvaia","arvaia","arvaia
> ","arvaia",
>"magli","magli","magli","magli","arvaia",
>"arvaia","arvaia","magli","magli","arvaia"
> )),
> trappola =
> as.factor(c("fmach__feromoni___vibrazione","pessl_elettronica",
>"fmach__feromoni___vibrazione","cymatrap__
> feromoni",
>"fmach__feromoni___vibrazione","cymatrap__
> feromoni",
>"fmach__feromoni___vibrazione","fmach__fer
> omoni___vibrazione",
>"cymatrap__feromoni","cymatrap__feromoni",
>"fmach__feromoni___vibrazione","fmach__fer
> omoni___vibrazione",
>"fmach__feromoni___vibrazione",
>"fmach__feromoni___vibrazione","cymatrap__
> feromoni",
>"fmach__feromoni___vibrazione","fmach__fer
> omoni___vibrazione",
>"cymatrap__feromoni","cymatrap__feromoni",
> "cymatrap__feromoni",
>"fmach__feromoni___vibrazione",
>"fmach__feromoni___vibrazione","fmach__fer
> omoni___vibrazione",
>"fmach__feromoni___vibrazione","cymatrap__
> feromoni",
>"fmach__feromoni___vibrazione","cymatrap__
> feromoni",
>"fmach__feromoni___vibrazione"))
> )
>
> # calculates the cumulative capture by type of trap + farm
> library(dplyr)
> vindicta_catture_cimici = vindicta_catture_cimici %>%
> group_by(trappola,azienda) %>% arrange(datetime) %>% mutate(cs =
> cumsum(catture))
> vindicta_catture_cimici <-
> group_by(vindicta_catture_cimici,trappola,azienda)
>
> # generates a matrix of the different irregular time-series of the
> cumulative values of capture per trap
> library(reshape2)
> library(xts)
> vindicta_catture_cimici_xts <-
> as.xts.data.table(data.table(dcast(vindicta_catture_cimici, datetime
> ~ trappola + azienda, value.var = "cs")))
>
> # calculates by interpolation the capture data at the end of each
> week, starting from May 10 (i.e. the week we started capturing)
> tseq <- seq(ISOdate(2021,5,10,0,0,0),
> end(vindicta_catture_cimici_xts), by = "week")
> vindicta_catture_cimici_xts_week <-
> na.approx(vindicta_catture_cimici_xts,xout=tseq)
> vindicta_catture_settimanali <-
> diff.xts(vindicta_catture_cimici_xts_week)
>
> library(TSstudio)
> ts_plot(vindicta_catture_settimanali)
Thank you
Enrico Gabrielli
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] aggregation of irregular interval time-series
sorry, i'm a bit of a bummer at copying and saving various pieces of
code
and I'm not very experienced with R
- I am trying to build a partnership with a local IT cooperative, but
this is a job that was not foreseen and it is not covered by budget, so
I am trying to solve it myself -
I do not want to ask in ml the ready turnkey solution, but some ideas
of what could be the set of packages and functions that could be right
for me, and if anyone has been faced with such a problem.
Thank you
Enrico Gabrielli
Il giorno mer, 09/06/2021 alle 14.47 -0700, Bert Gunter ha scritto:
> I have *not* followed this in any detail, but this line seems wrong:
>
> arvaia_catture_order <- arvaia_catture[order(arvaia_catture$tempo)]
> Perhaps it should be: arvaia_catture_order <-
> arvaia_catture[order(arvaia_catture$tempo), ] ## note the comma!
> If I am mistaken, just ignore and move on.
>
> Cheers,Bert
> Bert Gunter
>
> "The trouble with having an open mind is that people keep coming
> along and sticking things into it."
> -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )
>
>
> On Wed, Jun 9, 2021 at 2:41 PM Rui Barradas
> wrote:
> > Hello,
> >
> >
> >
> > I'm not getting your simple sum:
> >
> >
> >
> >
> >
> > arvaia_catture_order[, week := as.integer(format(tempo, "%U"))]
> >
> > aggregate(catture ~ week, arvaia_catture_order, sum)
> >
> > # week catture
> >
> > #1 19 15
> >
> > #2 20 5
> >
> > #3 21 78
> >
> > #4 22 120
> >
> >
> >
> >
> >
> > Can you explain your result better?
> >
> >
> >
> > Hope this helps,
> >
> >
> >
> > Rui Barradas
> >
> >
> >
> > Às 18:57 de 09/06/21, Enrico Gabrielli escreveu:
> >
> > > Hello
> >
> > > I just registered on the list.
> >
> > > I am an agricultural technician and I am collaborating on a
> > research
> >
> > > project on agroforestry and Brown Marmorated Stink Bug
> > (Halyomorpha
> >
> > > halys, abbreviated BMSB).
> >
> > >
> >
> > > Through kobotoolbox we are collecting data of catches in traps on
> >
> > > farms. Farms register inconsistently.
> >
> > > I am trying to use packages for irregular time series.
> >
> > >
> >
> > > Here an exemple
> >
> > > the data:
> >
> > > # variable monitoring time as register an import from kobotoolbox
> >
> > > tempo <- as.POSIXct(c("2021-05-29 17:00:00 UTC","2021-06-05
> > 10:52:00
> >
> > > UTC","2021-06-01 17:00:00 UTC","2021-05-16 08:34:00 UTC","2021-
> > 06-05
> >
> > > 17:00:00 UTC","2021-05-29 05:30:00 UTC","2021-05-23 06:30:00
> >
> > > UTC","2021-05-20 13:00:00 UTC","2021-05-15 12:09:00 UTC"))
> >
> > > # variable capture of BMSB
> >
> > > catture <- c(25,92,23,2,5,30,23,3,15)
> >
> > > # resulting table
> >
> > > library(data.table)
> >
> > > arvaia_catture <- data.table(tempo,catture)
> >
> > > # order by time
> >
> > > arvaia_catture_order <-
> > arvaia_catture[order(arvaia_catture$tempo)]
> >
> > >
> >
> > > the catches refer to an interval, which goes from the previous
> >
> > > monitoring up to the one recorded on the date
> >
> > > our aim is to calculate the weekly catch
> >
> > > also when a farmer, for example, enters the data on Thursday and
> >
> > > Tuesday
> >
> > > of the following week, on Tuesday in the trap he will find
> > individuals
> >
> > > who were also captured on Friday and Saturday, which formally are
> > to be
> >
> > > considered in the previous week.
> >
> > >
> >
> > > With the data of the example
> >
> > > the results (with a spreadsheet) is
> >
> > > two possibile solution
> >
> > > weeksimple SUM week right SUM week
> >
> > > 19 17 19,44027778
> >
> > > 20 26 52,9139
> >
> > > 21 55 46,4375integrand_2 <- function(x) {0.62 * (1/x)}
> >
> >
> >
> > integrate(integrand, lower = , upper = 5)
> >
> >
> >
> > > 22 120 99,2083
> >
> > > The right SUM week is right!
> >
> > >
> >
> > > I have made several attempts
> >
> > > with lubridate, padr, xts
> >
> > > but the last one seems interesting to me
> >
> > > with DTSg
> >
> > > doing
> >
> > > x_periodic <- alter(x,na.status = 'explicit',from="2021-05-15
> >
> > > 12:00:00",by="min")
> >
> > > colapply(x_periodic, fun = interpolateLinear)
> >
> > > I managed to create a vector with all interpolated hours
> >
> > > but with DTSg I still can't aggregate by week
> >
> > >
> >
> > > Has anyone on the list ever faced such a problem?
> >
> > >
> >
> > > Thank you
> >
> > >
> >
> >
> >
> > __
> >
> > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> >
> > https://stat.ethz.ch/mailman/listinfo/r-help
> >
> > PLEASE do read the posting guide
> > http://www.R-project.org/posting-guide.html
> >
> > and provide commented, minimal, self-contained, reproducible code.
> >
>
>
[[alternative HTML version deleted]]
__
R-help@r-proj
Re: [R] aggregation of irregular interval time-series
I have *not* followed this in any detail, but this line seems wrong:
arvaia_catture_order <- arvaia_catture[order(arvaia_catture$tempo)]
Perhaps it should be:
arvaia_catture_order <- arvaia_catture[order(arvaia_catture$tempo), ] ##
note the comma!
If I am mistaken, just ignore and move on.
Cheers,
Bert
Bert Gunter
"The trouble with having an open mind is that people keep coming along and
sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )
On Wed, Jun 9, 2021 at 2:41 PM Rui Barradas wrote:
> Hello,
>
> I'm not getting your simple sum:
>
>
> arvaia_catture_order[, week := as.integer(format(tempo, "%U"))]
> aggregate(catture ~ week, arvaia_catture_order, sum)
> # week catture
> #1 19 15
> #2 20 5
> #3 21 78
> #4 22 120
>
>
> Can you explain your result better?
>
> Hope this helps,
>
> Rui Barradas
>
> Às 18:57 de 09/06/21, Enrico Gabrielli escreveu:
> > Hello
> > I just registered on the list.
> > I am an agricultural technician and I am collaborating on a research
> > project on agroforestry and Brown Marmorated Stink Bug (Halyomorpha
> > halys, abbreviated BMSB).
> >
> > Through kobotoolbox we are collecting data of catches in traps on
> > farms. Farms register inconsistently.
> > I am trying to use packages for irregular time series.
> >
> > Here an exemple
> > the data:
> > # variable monitoring time as register an import from kobotoolbox
> > tempo <- as.POSIXct(c("2021-05-29 17:00:00 UTC","2021-06-05 10:52:00
> > UTC","2021-06-01 17:00:00 UTC","2021-05-16 08:34:00 UTC","2021-06-05
> > 17:00:00 UTC","2021-05-29 05:30:00 UTC","2021-05-23 06:30:00
> > UTC","2021-05-20 13:00:00 UTC","2021-05-15 12:09:00 UTC"))
> > # variable capture of BMSB
> > catture <- c(25,92,23,2,5,30,23,3,15)
> > # resulting table
> > library(data.table)
> > arvaia_catture <- data.table(tempo,catture)
> > # order by time
> > arvaia_catture_order <- arvaia_catture[order(arvaia_catture$tempo)]
> >
> > the catches refer to an interval, which goes from the previous
> > monitoring up to the one recorded on the date
> > our aim is to calculate the weekly catch
> > also when a farmer, for example, enters the data on Thursday and
> > Tuesday
> > of the following week, on Tuesday in the trap he will find individuals
> > who were also captured on Friday and Saturday, which formally are to be
> > considered in the previous week.
> >
> > With the data of the example
> > the results (with a spreadsheet) is
> > two possibile solution
> > weeksimple SUM week right SUM week
> > 19 17 19,44027778
> > 20 26 52,9139
> > 21 55 46,4375integrand_2 <- function(x) {0.62 * (1/x)}
>
> integrate(integrand, lower = , upper = 5)
>
> > 22 120 99,2083
> > The right SUM week is right!
> >
> > I have made several attempts
> > with lubridate, padr, xts
> > but the last one seems interesting to me
> > with DTSg
> > doing
> > x_periodic <- alter(x,na.status = 'explicit',from="2021-05-15
> > 12:00:00",by="min")
> > colapply(x_periodic, fun = interpolateLinear)
> > I managed to create a vector with all interpolated hours
> > but with DTSg I still can't aggregate by week
> >
> > Has anyone on the list ever faced such a problem?
> >
> > Thank you
> >
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] aggregation of irregular interval time-series
Hello,
I'm not getting your simple sum:
arvaia_catture_order[, week := as.integer(format(tempo, "%U"))]
aggregate(catture ~ week, arvaia_catture_order, sum)
# week catture
#1 19 15
#2 20 5
#3 21 78
#4 22 120
Can you explain your result better?
Hope this helps,
Rui Barradas
Às 18:57 de 09/06/21, Enrico Gabrielli escreveu:
Hello
I just registered on the list.
I am an agricultural technician and I am collaborating on a research
project on agroforestry and Brown Marmorated Stink Bug (Halyomorpha
halys, abbreviated BMSB).
Through kobotoolbox we are collecting data of catches in traps on
farms. Farms register inconsistently.
I am trying to use packages for irregular time series.
Here an exemple
the data:
# variable monitoring time as register an import from kobotoolbox
tempo <- as.POSIXct(c("2021-05-29 17:00:00 UTC","2021-06-05 10:52:00
UTC","2021-06-01 17:00:00 UTC","2021-05-16 08:34:00 UTC","2021-06-05
17:00:00 UTC","2021-05-29 05:30:00 UTC","2021-05-23 06:30:00
UTC","2021-05-20 13:00:00 UTC","2021-05-15 12:09:00 UTC"))
# variable capture of BMSB
catture <- c(25,92,23,2,5,30,23,3,15)
# resulting table
library(data.table)
arvaia_catture <- data.table(tempo,catture)
# order by time
arvaia_catture_order <- arvaia_catture[order(arvaia_catture$tempo)]
the catches refer to an interval, which goes from the previous
monitoring up to the one recorded on the date
our aim is to calculate the weekly catch
also when a farmer, for example, enters the data on Thursday and
Tuesday
of the following week, on Tuesday in the trap he will find individuals
who were also captured on Friday and Saturday, which formally are to be
considered in the previous week.
With the data of the example
the results (with a spreadsheet) is
two possibile solution
weeksimple SUM week right SUM week
19 17 19,44027778
20 26 52,9139
21 55 46,4375integrand_2 <- function(x) {0.62 * (1/x)}
integrate(integrand, lower = , upper = 5)
22 120 99,2083
The right SUM week is right!
I have made several attempts
with lubridate, padr, xts
but the last one seems interesting to me
with DTSg
doing
x_periodic <- alter(x,na.status = 'explicit',from="2021-05-15
12:00:00",by="min")
colapply(x_periodic, fun = interpolateLinear)
I managed to create a vector with all interpolated hours
but with DTSg I still can't aggregate by week
Has anyone on the list ever faced such a problem?
Thank you
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] aggregation of irregular interval time-series
Hello
I just registered on the list.
I am an agricultural technician and I am collaborating on a research
project on agroforestry and Brown Marmorated Stink Bug (Halyomorpha
halys, abbreviated BMSB).
Through kobotoolbox we are collecting data of catches in traps on
farms. Farms register inconsistently.
I am trying to use packages for irregular time series.
Here an exemple
the data:
# variable monitoring time as register an import from kobotoolbox
tempo <- as.POSIXct(c("2021-05-29 17:00:00 UTC","2021-06-05 10:52:00
UTC","2021-06-01 17:00:00 UTC","2021-05-16 08:34:00 UTC","2021-06-05
17:00:00 UTC","2021-05-29 05:30:00 UTC","2021-05-23 06:30:00
UTC","2021-05-20 13:00:00 UTC","2021-05-15 12:09:00 UTC"))
# variable capture of BMSB
catture <- c(25,92,23,2,5,30,23,3,15)
# resulting table
library(data.table)
arvaia_catture <- data.table(tempo,catture)
# order by time
arvaia_catture_order <- arvaia_catture[order(arvaia_catture$tempo)]
the catches refer to an interval, which goes from the previous
monitoring up to the one recorded on the date
our aim is to calculate the weekly catch
also when a farmer, for example, enters the data on Thursday and
Tuesday
of the following week, on Tuesday in the trap he will find individuals
who were also captured on Friday and Saturday, which formally are to be
considered in the previous week.
With the data of the example
the results (with a spreadsheet) is
two possibile solution
weeksimple SUM week right SUM week
19 17 19,44027778
20 26 52,9139
21 55 46,4375
22 120 99,2083
The right SUM week is right!
I have made several attempts
with lubridate, padr, xts
but the last one seems interesting to me
with DTSg
doing
x_periodic <- alter(x,na.status = 'explicit',from="2021-05-15
12:00:00",by="min")
colapply(x_periodic, fun = interpolateLinear)
I managed to create a vector with all interpolated hours
but with DTSg I still can't aggregate by week
Has anyone on the list ever faced such a problem?
Thank you
--
Perito agrario Enrico Gabrielli
Tessera n. 633 Collegio Periti agrari prov. Di Modena
Biblioteca agricoltura: https://www.zotero.org/groups/aplomb/
https://it.linkedin.com/pub/enrico-gabrielli/9a/186/159
https://enricogabrielli76.wordpress.com/
https://www.inaturalist.org/observations/bonushenricus
skype: enricogabrielli (enricogabrielli76.per...@gmail.com)
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
