Re: [R] Time intervals is converted into seconds after converting list of dfs into a single Df.
Many thanks Bert for being so cooperative.
Deviding the data into small bites would be
a good suggestion but I will wait first to see
If someone else may have another idea.
Many thanks
From: Bert Gunter
Sent: 24 December 2019 21:03:56
To: Allaisone 1
Cc: Patrick (Malone Quantitative) ;
r-help@r-project.org
Subject: Re: [R] Time intervals is converted into seconds after converting list
of dfs into a single Df.
1. "Similar" or "same" column names. The former is probably not going to work.
2. Manipulations with data frames can consume a lot of memory. rbinding 8000
data frames is likely to be very slow with lots of time swapping memory
around(???). Perhaps try taking smaller bites (say 1000 at a time) and then
combining them. Or have you already tried this? If you do wish to do this, wait
to give experts a chance to tell you that my suggestion is completely useless
before you attempt it.
3. I'll let someone else resolve your dates problem, as I have never used
lubridate.
Bert Gunter
"The trouble with having an open mind is that people keep coming along and
sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )
On Tue, Dec 24, 2019 at 12:38 PM Allaisone 1
mailto:allaiso...@hotmail.com>> wrote:
Hi dear Patrick ,
Thanks for your replay. Below is a reproducible example . First, I generated
two similar Dfs with one column contains the interval. Then, I put the 2 dfs
in a list. Now, converting this list into df provides different results
depending on the code. See below for more details.
# dataframe 1
id <- c(1,1)
dates1 <- c("2010/2/4","2011/2/4")
dates2 <- c("2010/9/4","2011/1/1")
df1 <- data.frame(id,dates1,dates2)
df1[,2] <- as.Date(df1[,2])
df1[,3] <- as.Date(df1[,3])
df1$interaction <-
intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))
# Dataframe 2
id <- c(2,2)
dates1 <- c("2010/1/4","2011/2/4")
dates2 <- c("2010/10/4","2011/1/16")
df2 <- data.frame(id,dates1,dates2)
df2[,2] <- as.Date(df1[,2])
df2[,3] <- as.Date(df1[,3])
df2$interaction <-
intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))
# 2 datframes in a list :
ListOfDFs <- list(df1,df2)
# Convert list of Dfs into a single df :-
SingDF <- ldply( ListOfDFs,data.frame)
# The interval has been converted into numbers which is not what I want.
#but trying this code :
SingDF <- do.call(rbind,ListOfDFs)
# It works perfectly but only with this example as we have only 2
datframes. Howver, in my actual data I have around 8000 datframes. Applying
this code to it , make R code freezes and I waited for many hours but it still
freezes with no results generated.
Could anyone please suggest any alternative syntax or modifications to the
codes above?
Kind Regards
Sent from Outlook
From: Patrick (Malone Quantitative)
mailto:mal...@malonequantitative.com>>
Sent: 24 December 2019 17:01:59
To: Allaisone 1 mailto:allaiso...@hotmail.com>>
Cc: r-help@r-project.org
mailto:r-help@r-project.org>>
Subject: Re: [R] Time intervals is converted into seconds after converting list
of dfs into a single Df.
You didn't provide a reproducible example for testing (or post in
plain text), but lubridate has an as.interval() function. You'll need
to be able to extract the start time, though, for use in the function.
On Tue, Dec 24, 2019 at 11:54 AM Allaisone 1
mailto:allaiso...@hotmail.com>> wrote:
>
>
> Hi dear group ,
>
> I have list of datframes with similar column names. I want to rebind all
> dataframes so I have a single dataframe. One of the column's in each df is of
> 'interval' time class which was generated from 'lubridate' package.
>
> The problem is that when I convert the list of dfs into a single df using any
> of the below codes :
>
> Library(plyr)
> MySingleDf <- ldply(MyListOfDfs, data.frame)
> Or
> MySingleDf <- ldply(MyListOfDfs, rbind)
> Or
> MySingleDf <- rebind. fill (MyListOfDfs)
>
> What heppens is that time intervals which looks like : 2010-4-5
> UTC--2011-7-9 UTC is converted into a single numeric value which seems to be
> the difference between the 2 dates in seconds.
>
> When I use :
> MySingleDf <- do.call ("rbind",MyListOfDfs)
>
> The code is freezes and it shows like of the data are being analysed but no
> result. I have used this code previously for the same purpose but with
> another datse and it works perfectly.
>
> What I want to see is that time intervals are shown as they are but not
> converted into seconds.
>
> Could you please suggest any alternative syntax or modifications to my codes ?
>
> Thank you so much in advance
>
> Regards
>
>
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list -- To
> UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
>
Re: [R] Time intervals is converted into seconds after converting list of dfs into a single Df.
Perhaps some modification of
masterList <- list()
for( dfnum in seq_along(ListOfDFs)){
masterList <- rbind(masterList, ListOfDFs[[dfnum]])
}
masterList
#---
id dates1 dates2 interaction
1 1 2010-02-04 2010-09-04 2010-09-04 UTC--2011-01-01 UTC
2 1 2011-02-04 2011-01-01 2010-09-04 UTC--2011-01-01 UTC
3 2 2010-02-04 2010-09-04 2010-09-04 UTC--2011-01-01 UTC
4 2 2011-02-04 2011-01-01 2010-09-04 UTC--2011-01-01
You could add features to the for-loop such as printing a message to the
console every 100 dfs or perhaps garbage collection although that should
be handled automagically. Messages would probably reassure you that the
process was not "hanging". (My suspicion is that the process was
continuing but you were just too impatient.)
Note; you are posting in HTML and including non-printing characters in
you code.
--
David
On 12/24/19 10:53 AM, Allaisone 1 wrote:
Hi dear Patrick ,
Thanks for your replay. Below is a reproducible example . First, I generated
two similar Dfs with one column contains the interval. Then, I put the 2 dfs
in a list. Now, converting this list into df provides different results
depending on the code. See below for more details.
# dataframe 1
id <- c(1,1)
dates1 <- c("2010/2/4","2011/2/4")
dates2 <- c("2010/9/4","2011/1/1")
df1 <- data.frame(id,dates1,dates2)
df1[,2] <- as.Date(df1[,2])
df1[,3] <- as.Date(df1[,3])
df1$interaction <-
intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))
# Dataframe 2
id <- c(2,2)
dates1 <- c("2010/1/4","2011/2/4")
dates2 <- c("2010/10/4","2011/1/16")
df2 <- data.frame(id,dates1,dates2)
df2[,2] <- as.Date(df1[,2])
df2[,3] <- as.Date(df1[,3])
df2$interaction <-
intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))
# 2 datframes in a list :
ListOfDFs <- list(df1,df2)
# Convert list of Dfs into a single df :-
SingDF <- ldply( ListOfDFs,data.frame)
# The interval has been converted into numbers which is not what I
want.
#but trying this code :
SingDF <- do.call(rbind,ListOfDFs)
# It works perfectly but only with this example as we have only 2
datframes. Howver, in my actual data I have around 8000 datframes. Applying
this code to it , make R code freezes and I waited for many hours but it still
freezes with no results generated.
Could anyone please suggest any alternative syntax or modifications to the
codes above?
Kind Regards
Sent from Outlook
From: Patrick (Malone Quantitative)
Sent: 24 December 2019 17:01:59
To: Allaisone 1
Cc: r-help@r-project.org
Subject: Re: [R] Time intervals is converted into seconds after converting list
of dfs into a single Df.
You didn't provide a reproducible example for testing (or post in
plain text), but lubridate has an as.interval() function. You'll need
to be able to extract the start time, though, for use in the function.
On Tue, Dec 24, 2019 at 11:54 AM Allaisone 1 wrote:
Hi dear group ,
I have list of datframes with similar column names. I want to rebind all
dataframes so I have a single dataframe. One of the column's in each df is of
'interval' time class which was generated from 'lubridate' package.
The problem is that when I convert the list of dfs into a single df using any
of the below codes :
Library(plyr)
MySingleDf <- ldply(MyListOfDfs, data.frame)
Or
MySingleDf <- ldply(MyListOfDfs, rbind)
Or
MySingleDf <- rebind. fill (MyListOfDfs)
What heppens is that time intervals which looks like : 2010-4-5 UTC--2011-7-9
UTC is converted into a single numeric value which seems to be the difference
between the 2 dates in seconds.
When I use :
MySingleDf <- do.call ("rbind",MyListOfDfs)
The code is freezes and it shows like of the data are being analysed but no
result. I have used this code previously for the same purpose but with another
datse and it works perfectly.
What I want to see is that time intervals are shown as they are but not
converted into seconds.
Could you please suggest any alternative syntax or modifications to my codes ?
Thank you so much in advance
Regards
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Time intervals is converted into seconds after converting list of dfs into a single Df.
1. "Similar" or "same" column names. The former is probably not going to
work.
2. Manipulations with data frames can consume a lot of memory. rbinding
8000 data frames is likely to be very slow with lots of time swapping
memory around(???). Perhaps try taking smaller bites (say 1000 at a time)
and then combining them. Or have you already tried this? If you do wish to
do this, wait to give experts a chance to tell you that my suggestion is
completely useless before you attempt it.
3. I'll let someone else resolve your dates problem, as I have never used
lubridate.
Bert Gunter
"The trouble with having an open mind is that people keep coming along and
sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )
On Tue, Dec 24, 2019 at 12:38 PM Allaisone 1 wrote:
> Hi dear Patrick ,
>
> Thanks for your replay. Below is a reproducible example . First, I
> generated two similar Dfs with one column contains the interval. Then, I
> put the 2 dfs in a list. Now, converting this list into df provides
> different results depending on the code. See below for more details.
>
>
> # dataframe 1
>
> id <- c(1,1)
>
> dates1 <- c("2010/2/4","2011/2/4")
>
> dates2 <- c("2010/9/4","2011/1/1")
>
> df1 <- data.frame(id,dates1,dates2)
>
> df1[,2] <- as.Date(df1[,2])
>
> df1[,3] <- as.Date(df1[,3])
>
> df1$interaction <-
> intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))
>
>
>
> # Dataframe 2
>
> id <- c(2,2)
>
> dates1 <- c("2010/1/4","2011/2/4")
>
> dates2 <- c("2010/10/4","2011/1/16")
>
> df2 <- data.frame(id,dates1,dates2)
>
> df2[,2] <- as.Date(df1[,2])
>
> df2[,3] <- as.Date(df1[,3])
>
>
> df2$interaction <-
> intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))
>
>
>
> # 2 datframes in a list :
>
> ListOfDFs <- list(df1,df2)
>
> # Convert list of Dfs into a single df :-
>
> SingDF <- ldply( ListOfDFs,data.frame)
>
># The interval has been converted into numbers which is not what I
> want.
>
>#but trying this code :
> SingDF <- do.call(rbind,ListOfDFs)
>
># It works perfectly but only with this example as we have only 2
> datframes. Howver, in my actual data I have around 8000 datframes. Applying
> this code to it , make R code freezes and I waited for many hours but it
> still freezes with no results generated.
>
> Could anyone please suggest any alternative syntax or modifications to
> the codes above?
>
> Kind Regards
>
>
>
>
> Sent from Outlook
>
> From: Patrick (Malone Quantitative)
> Sent: 24 December 2019 17:01:59
> To: Allaisone 1
> Cc: r-help@r-project.org
> Subject: Re: [R] Time intervals is converted into seconds after converting
> list of dfs into a single Df.
>
> You didn't provide a reproducible example for testing (or post in
> plain text), but lubridate has an as.interval() function. You'll need
> to be able to extract the start time, though, for use in the function.
>
> On Tue, Dec 24, 2019 at 11:54 AM Allaisone 1
> wrote:
> >
> >
> > Hi dear group ,
> >
> > I have list of datframes with similar column names. I want to rebind all
> dataframes so I have a single dataframe. One of the column's in each df is
> of 'interval' time class which was generated from 'lubridate' package.
> >
> > The problem is that when I convert the list of dfs into a single df
> using any of the below codes :
> >
> > Library(plyr)
> > MySingleDf <- ldply(MyListOfDfs, data.frame)
> > Or
> > MySingleDf <- ldply(MyListOfDfs, rbind)
> > Or
> > MySingleDf <- rebind. fill (MyListOfDfs)
> >
> > What heppens is that time intervals which looks like : 2010-4-5
> UTC--2011-7-9 UTC is converted into a single numeric value which seems to
> be the difference between the 2 dates in seconds.
> >
> > When I use :
> > MySingleDf <- do.call ("rbind",MyListOfDfs)
> >
> > The code is freezes and it shows like of the data are being analysed but
> no result. I have used this code previously for the same purpose but with
> another datse and it works perfectly.
> >
> > What I want to see is that time intervals are shown as they are but not
> converted into seconds.
> >
> > Could you please suggest any alternative syntax or modifications to my
> codes ?
> >
> > Thank you so much in advance
> >
> > Regards
> >
> >
> >
> > [[alternative HTML version deleted]]
> >
> > __
> > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal,
Re: [R] Time intervals is converted into seconds after converting list of dfs into a single Df.
Hi dear Patrick ,
Thanks for your replay. Below is a reproducible example . First, I generated
two similar Dfs with one column contains the interval. Then, I put the 2 dfs
in a list. Now, converting this list into df provides different results
depending on the code. See below for more details.
# dataframe 1
id <- c(1,1)
dates1 <- c("2010/2/4","2011/2/4")
dates2 <- c("2010/9/4","2011/1/1")
df1 <- data.frame(id,dates1,dates2)
df1[,2] <- as.Date(df1[,2])
df1[,3] <- as.Date(df1[,3])
df1$interaction <-
intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))
# Dataframe 2
id <- c(2,2)
dates1 <- c("2010/1/4","2011/2/4")
dates2 <- c("2010/10/4","2011/1/16")
df2 <- data.frame(id,dates1,dates2)
df2[,2] <- as.Date(df1[,2])
df2[,3] <- as.Date(df1[,3])
df2$interaction <-
intersect(interval(df1[1,2],df1[2,2]),interval(df1[1,3],df1[2,3]))
# 2 datframes in a list :
ListOfDFs <- list(df1,df2)
# Convert list of Dfs into a single df :-
SingDF <- ldply( ListOfDFs,data.frame)
# The interval has been converted into numbers which is not what I want.
#but trying this code :
SingDF <- do.call(rbind,ListOfDFs)
# It works perfectly but only with this example as we have only 2
datframes. Howver, in my actual data I have around 8000 datframes. Applying
this code to it , make R code freezes and I waited for many hours but it still
freezes with no results generated.
Could anyone please suggest any alternative syntax or modifications to the
codes above?
Kind Regards
Sent from Outlook
From: Patrick (Malone Quantitative)
Sent: 24 December 2019 17:01:59
To: Allaisone 1
Cc: r-help@r-project.org
Subject: Re: [R] Time intervals is converted into seconds after converting list
of dfs into a single Df.
You didn't provide a reproducible example for testing (or post in
plain text), but lubridate has an as.interval() function. You'll need
to be able to extract the start time, though, for use in the function.
On Tue, Dec 24, 2019 at 11:54 AM Allaisone 1 wrote:
>
>
> Hi dear group ,
>
> I have list of datframes with similar column names. I want to rebind all
> dataframes so I have a single dataframe. One of the column's in each df is of
> 'interval' time class which was generated from 'lubridate' package.
>
> The problem is that when I convert the list of dfs into a single df using any
> of the below codes :
>
> Library(plyr)
> MySingleDf <- ldply(MyListOfDfs, data.frame)
> Or
> MySingleDf <- ldply(MyListOfDfs, rbind)
> Or
> MySingleDf <- rebind. fill (MyListOfDfs)
>
> What heppens is that time intervals which looks like : 2010-4-5
> UTC--2011-7-9 UTC is converted into a single numeric value which seems to be
> the difference between the 2 dates in seconds.
>
> When I use :
> MySingleDf <- do.call ("rbind",MyListOfDfs)
>
> The code is freezes and it shows like of the data are being analysed but no
> result. I have used this code previously for the same purpose but with
> another datse and it works perfectly.
>
> What I want to see is that time intervals are shown as they are but not
> converted into seconds.
>
> Could you please suggest any alternative syntax or modifications to my codes ?
>
> Thank you so much in advance
>
> Regards
>
>
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Time intervals is converted into seconds after converting list of dfs into a single Df.
You didn't provide a reproducible example for testing (or post in
plain text), but lubridate has an as.interval() function. You'll need
to be able to extract the start time, though, for use in the function.
On Tue, Dec 24, 2019 at 11:54 AM Allaisone 1 wrote:
>
>
> Hi dear group ,
>
> I have list of datframes with similar column names. I want to rebind all
> dataframes so I have a single dataframe. One of the column's in each df is of
> 'interval' time class which was generated from 'lubridate' package.
>
> The problem is that when I convert the list of dfs into a single df using any
> of the below codes :
>
> Library(plyr)
> MySingleDf <- ldply(MyListOfDfs, data.frame)
> Or
> MySingleDf <- ldply(MyListOfDfs, rbind)
> Or
> MySingleDf <- rebind. fill (MyListOfDfs)
>
> What heppens is that time intervals which looks like : 2010-4-5
> UTC--2011-7-9 UTC is converted into a single numeric value which seems to be
> the difference between the 2 dates in seconds.
>
> When I use :
> MySingleDf <- do.call ("rbind",MyListOfDfs)
>
> The code is freezes and it shows like of the data are being analysed but no
> result. I have used this code previously for the same purpose but with
> another datse and it works perfectly.
>
> What I want to see is that time intervals are shown as they are but not
> converted into seconds.
>
> Could you please suggest any alternative syntax or modifications to my codes ?
>
> Thank you so much in advance
>
> Regards
>
>
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] Time intervals is converted into seconds after converting list of dfs into a single Df.
Hi dear group ,
I have list of datframes with similar column names. I want to rebind all
dataframes so I have a single dataframe. One of the column's in each df is of
'interval' time class which was generated from 'lubridate' package.
The problem is that when I convert the list of dfs into a single df using any
of the below codes :
Library(plyr)
MySingleDf <- ldply(MyListOfDfs, data.frame)
Or
MySingleDf <- ldply(MyListOfDfs, rbind)
Or
MySingleDf <- rebind. fill (MyListOfDfs)
What heppens is that time intervals which looks like : 2010-4-5 UTC--2011-7-9
UTC is converted into a single numeric value which seems to be the difference
between the 2 dates in seconds.
When I use :
MySingleDf <- do.call ("rbind",MyListOfDfs)
The code is freezes and it shows like of the data are being analysed but no
result. I have used this code previously for the same purpose but with another
datse and it works perfectly.
What I want to see is that time intervals are shown as they are but not
converted into seconds.
Could you please suggest any alternative syntax or modifications to my codes ?
Thank you so much in advance
Regards
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
